task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-vswkg-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-vswkg-kubeconfig [get-kubeconfig] Found admin password secret: cluster-vswkg-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-vswkg-password [get-kubeconfig] API Server URL: https://a15f160cb9fc049e090daca22bfeb390-08bf69e8cbd20e94.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.d9a74490-1df8-4f47-b68c-b2673f0f1355.prod.konfluxeaas.com [clone-repo] early-gate-onboard-1781251679 [clone-repo] https://github.com/opendatahub-io/kserve [clone-repo] Cloning into '/workspace/source'... [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:41e9c610ce929fc56f4076f97bf42d251460cf8b6a90f49e0a84e5b92946522f [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:41e9c610ce929fc56f4076f97bf42d251460cf8b6a90f49e0a84e5b92946522f [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:9fb811904a32cf8c473f3294497ae3de2880b3821ebeb2ef9ae45e81dcbf43d6 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:9fb811904a32cf8c473f3294497ae3de2880b3821ebeb2ef9ae45e81dcbf43d6 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:c92ab94c818c88d4324be5bb5a3d9bda3e4173010dc47851ff3051c5a06f1614 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:c92ab94c818c88d4324be5bb5a3d9bda3e4173010dc47851ff3051c5a06f1614 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:bfa2ac28e73bf5929baf7f3bcb5adf0e285b81b7c3ac6c3c85c08cdb5443e73f [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:bfa2ac28e73bf5929baf7f3bcb5adf0e285b81b7c3ac6c3c85c08cdb5443e73f [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling' 2 llm-d [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1619 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1619 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1619 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.79s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 461ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.121.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==0.49.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 45ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a15f160cb9fc049e090daca22bfeb390-08bf69e8cbd20e94.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.18.1-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-674595557b-rbf4c 1/1 Running 0 42s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-674595557b-rbf4c condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-68788fbd9b-bsktr 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-68788fbd9b-bsktr condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-c6d879546-v7nv9 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-c6d879546-v7nv9 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:9fb811904a32cf8c473f3294497ae3de2880b3821ebeb2ef9ae45e81dcbf43d6 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:bfa2ac28e73bf5929baf7f3bcb5adf0e285b81b7c3ac6c3c85c08cdb5443e73f [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:41e9c610ce929fc56f4076f97bf42d251460cf8b6a90f49e0a84e5b92946522f [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:c92ab94c818c88d4324be5bb5a3d9bda3e4173010dc47851ff3051c5a06f1614 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-58d9dd6777-f4fnv 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-58d9dd6777-f4fnv condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.20) [e2e-llm-inference-service] 🎯 Server version (4.21.20) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (15/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (20/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (25/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (30/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.19.0 found, but not yet Succeeded (Phase: Installing). Waiting... (35/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.19.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-d9kj2 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-d9kj2 condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-75c67f8887-qbmcr 1/1 Running 0 7s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-75c67f8887-qbmcr condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd 1/1 Running 0 7s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.0 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.0 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.0 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d75c86569-cxdzr 1/1 Running 0 76s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d75c86569-cxdzr condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d75c86569-cxdzr 1/1 Running 0 86s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d75c86569-cxdzr condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-6c7654bd99-fz97p" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-6c7654bd99-m8vkw condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] Warning: resource clusterroles/metrics-reader is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader configured [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.20 [e2e-llm-inference-service] Kubernetes Version: v1.34.8 [e2e-llm-inference-service] ClusterVersion desired: 4.21.20 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.20 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.0 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.0 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.2.0 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.0 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.18.1-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.0 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.2.0 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.0 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.0 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.2.0 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.0 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.0 Succeeded [e2e-llm-inference-service] servicemeshoperator3.v3.2.0 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.0 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.0 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.0 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:9fb811904a32cf8c473f3294497ae3de2880b3821ebeb2ef9ae45e81dcbf43d6 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:76ff0b6e3df81e33631a0dbe33a828950976ec44d6d86fb6fe1d9439e2a0af73 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:db80d75c1c3cc0873582c64e891323ef8d6f60440751b48a8bc5158495a15007 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:bfa2ac28e73bf5929baf7f3bcb5adf0e285b81b7c3ac6c3c85c08cdb5443e73f [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:bc551c48c50671c85a287b53bce25eba8b30a5aa71f8fa6a7b7c8014d3675552 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'anyio': '4.9.0', 'httpx': '0.30.0', 'xdist': '3.6.1', 'json-report': '1.5.0', 'metadata': '3.1.1', 'cov': '5.0.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-0.23.8, anyio-4.9.0, httpx-0.30.0, xdist-3.6.1, json-report-1.5.0, metadata-3.1.1, cov-5.0.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [35 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] 2026-06-15 06:01:44.666 6183 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-06-15 06:01:44.666 6186 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-06-15 06:01:44.681 6183 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:01:44.681 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:01:44.712 6183 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-06-15 06:01:44.712 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-06-15 06:01:44.724 6183 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-06-15 06:01:44.724 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-06-15 06:02:18.534 6183 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:02:18.534 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:02:18.583 6183 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-06-15 06:02:18.583 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-06-15 06:23:44.767 6183 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:23:44.767 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:23:44.812 6183 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-06-15 06:23:44.812 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-06-15 06:29:59.884 6183 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:29:59.884 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:29:59.935 6183 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-06-15 06:29:59.935 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-06-15 06:29:59.936 6183 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:29:59.936 6183 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:29:59.939 6183 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-06-15 06:29:59.939 6183 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-06-15 06:29:59.948 6183 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-06-15 06:29:59.948 6183 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-06-15 06:29:59.949 6183 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:29:59.949 6183 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:29:59.952 6183 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-06-15 06:29:59.952 6183 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-06-15 06:29:59.961 6183 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-06-15 06:29:59.961 6183 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-06-15 06:52:11.527 6183 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:52:11.527 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:52:11.558 6183 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-06-15 06:52:11.558 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-06-15 06:52:11.565 6183 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-06-15 06:52:11.565 6183 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-06-15 06:52:11.565 6183 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:52:11.565 6183 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:52:11.568 6183 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-06-15 06:52:11.568 6183 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-06-15 06:52:11.576 6183 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-06-15 06:52:11.576 6183 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-06-15 06:52:11.576 6183 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:52:11.576 6183 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-06-15 06:52:11.581 6183 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-06-15 06:52:11.581 6183 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-06-15 06:52:11.591 6183 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-06-15 06:52:11.591 6183 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] __________ test_llm_auth_enabled_requires_token[auth-enabled-default] __________ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name=... {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.auth [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="auth-enabled-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="auth-enabled-default", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_auth_enabled_requires_token(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] """ [e2e-llm-inference-service] Test that when auth is enabled (default): [e2e-llm-inference-service] - Requests WITH valid token succeed [e2e-llm-inference-service] - Requests WITHOUT token are rejected (401/403) [e2e-llm-inference-service] """ [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] sa_name = f"{service_name}-test-sa" [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Enable auth for this test [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "true" [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Create LLMInferenceService [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_auth.py:275: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-06-15T06:03:08.552722', start_time = 1781503388.5530953 [e2e-llm-inference-service] duration = 900.3913719654083, timestamp_end = '2026-06-15T06:18:08.944471' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....-auth-enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f1922fbafc0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1110: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-managed-auth-enabled-tes-210a8f79 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-managed-auth-enabled-tes-210a8f79 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-managed-auth-enabled-tes-210a8f79 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-single-cpu-auth-enable-a18fd8e2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-auth-enable-a18fd8e2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-auth-enable-a18fd8e2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig model-fb-opt-125m-auth-enabled-89f54b63 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-auth-enabled-89f54b63 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-auth-enabled-89f54b63 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_auth_enabled_requires_token] [2026-06-15T06:03:08.494351] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='auth-enabled-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-enabled-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-enabled-tes-210a8f79'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-06-15T06:03:08.507452] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'true'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-enabled-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-enabled-tes-210a8f79'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-06-15T06:03:08.552505] end - ✅ in 0.045s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-06-15T06:03:08.552722] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'true'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-enabled-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-enabled-tes-210a8f79'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-enable-a18fd8e2'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-enabled-89f54b63'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:03:12Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/auth-enabled-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:03:12Z', 'message': 'Inference Pool kserve-ci-e2e-test/auth-enabled-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:03:12Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:12Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:03:12Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/auth-enabled-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:03:12Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/auth-enabled-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:03:12Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:12Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1130 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-06-15T06:18:08.944471] end - ❌ 900.391s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_auth.py:345 ❌ ERROR: Failed test for auth-enabled-test: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1151 🔍 # Diagnostics for 'auth-enabled-test' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1152 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1153 # LLMInferenceService auth-enabled-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1156 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'true' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:08Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:08Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:08Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:03:44Z' [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '24863' [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-auth-enabled-tes-210a8f79 [e2e-llm-inference-service] - name: workload-single-cpu-auth-enable-a18fd8e2 [e2e-llm-inference-service] - name: model-fb-opt-125m-auth-enabled-89f54b63 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/auth-enabled-test [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/auth-enabled-test [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:44Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:44Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/auth-enabled-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-85d86d876c-vrqhw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" in 3.371s (3.371s including waiting). Image size: 299992506 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6c5d597fbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-85d86d876c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-779977f94c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.286s (1.286s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:07:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7bc88f48bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-67h82 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.023s (1.023s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-67h82 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-h6wcn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.38:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-74dcd66d7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5c556785f6 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.707s (2.707s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:33 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-75dcfd69c9-dh6qf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-75dcfd69c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:18 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-96f8b89cb-j7r99 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-96f8b89cb-j7r99 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-9c4c7855f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-96f8b89cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.082s (1.082s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 951ms (951ms including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.592s (30.592s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.034s (1.034s including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 31.996s (31.996s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.34:8082/healthz": dial tcp 10.133.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884fbb from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5d7479f884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod auth-enabled-test-kserve-85d86d876c-vrqhw (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:03:15.626 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 06:03:15.626 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_1111191a-3b3e-4255-a6bd-5135ceb67b34'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_d0f306dd-f79d-4df9-840d-b060d83ffd51'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_5814ac9f-73d0-416e-8611-eb4d39e94238'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e9b7afe4-06d4-4f9a-a33f-7bf032ab107d'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c1c3694f-15ff-412c-acd5-ec529b489425', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 15 Jun 2026 06:18:09 GMT', 'Content-Length': '223'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"auth-enabled-test-kserve-85d86d876c-vrqhw\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:03:12.560 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 06:03:12.560 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-06-15 06:03:12.560 1 storage.initializer INFO [kserve_storage.py:download():169] Allow patterns: ['tokenizer.json', 'tokenizer_config.json', 'special_tokens_map.json', 'vocab.json', 'merges.txt', 'config.json', 'generation_config.json'] [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_1ca9a00f-3a64-4168-81cb-94daba29ea2a'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ee1fc6c3-1076-4630-a157-95f9ff1944a6'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_eac72200-bd7b-4459-b575-c2d6072c9aff'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_f35ddd34-fd14-4dde-9cb3-74af3d5fd83e'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_45581c4c-50ce-468d-b60b-dc5d048397b5'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_63f10104-ca63-4fb9-b214-70ccf7ee5c99'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-06-15 06:03:12.944 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 06:03:12.944 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 0.3842706520000547 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":"2026-06-15T06:03:13Z","logger":"setup","caller":"runner/runner.go:150","msg":"GIE build","commit-sha":"","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:13Z","logger":"setup","caller":"runner/runner.go:169","msg":"Flags processed","flags":{"cache-info-metric":"vllm:cache_config_info","cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"kv-cache-usage-percentage-metric":"vllm:kv_cache_usage_perc","lora-info-metric":"vllm:lora_requests_info","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"auth-enabled-test-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"total-queued-requests-metric":"vllm:num_requests_waiting","total-running-requests-metric":"vllm:num_requests_running","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:13Z","logger":"setup.trace","caller":"tracing/telemetry.go:131","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:13Z","caller":"loader/configloader.go:65","msg":"Loaded raw configuration","config":"{FeatureGates: {}, Plugins: [{/single-profile-handler} {/queue-scorer} {/prefix-cache-scorer} {/max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"prefix/plugin.go:203","msg":"BlockSize is not positive, using default value","default":16} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"prefix/plugin.go:213","msg":"PrefixCachePlugin initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":256,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"loader/configloader.go:98","msg":"Effective configuration loaded","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","kind":"EndpointPickerConfig"},"configError":"got runtime.Object without object metadata: {FeatureGates: {}, Plugins: [{single-profile-handler/single-profile-handler} {queue-scorer/queue-scorer} {prefix-cache-scorer/prefix-cache-scorer} {max-score-picker/max-score-picker} {fcfs-ordering-policy/fcfs-ordering-policy} {global-strict-fairness-policy/global-strict-fairness-policy}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"runner/runner.go:549","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","logger":"setup","caller":"runner/runner.go:301","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","logger":"setup","caller":"runner/runner.go:315","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","logger":"setup.SaturationDetector","caller":"utilizationdetector/detector.go:70","msg":"Creating new SaturationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","logger":"setup","caller":"runner/runner.go:350","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","logger":"setup","caller":"runner/runner.go:644","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","logger":"setup","caller":"runner/runner.go:209","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:03:14Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:14Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"auth-enabled-test-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"auth-enabled-test-inference-pool","reconcileID":"11e0a996-8c0f-4de1-ae51-8926fee25333","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:03:17Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"auth-enabled-test-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"auth-enabled-test-inference-pool","reconcileID":"35b179c4-3eb2-4309-b269-7629eadf12f0","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:07:11Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"0a6ed530-fee4-4d61-af13-446a6a75ec8f"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:11Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"0a6ed530-fee4-4d61-af13-446a6a75ec8f","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:11Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"0a6ed530-fee4-4d61-af13-446a6a75ec8f","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:07:11Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"7717fe34-ece0-4e5f-ba19-c52f4ad81d6a"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:11Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"7717fe34-ece0-4e5f-ba19-c52f4ad81d6a","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:11Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"7717fe34-ece0-4e5f-ba19-c52f4ad81d6a","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:07:15Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"a1cec459-8a51-42c8-b592-7f9b68e8ef87"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:15Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"a1cec459-8a51-42c8-b592-7f9b68e8ef87","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:15Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"a1cec459-8a51-42c8-b592-7f9b68e8ef87","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:07:23Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"f2b1596f-d4ad-4dab-a8b9-d5ee9a582615"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:23Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"f2b1596f-d4ad-4dab-a8b9-d5ee9a582615","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:23Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"f2b1596f-d4ad-4dab-a8b9-d5ee9a582615","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:07:39Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"d86db5e3-6a6a-4baf-b948-1d614b932fd7"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:39Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"d86db5e3-6a6a-4baf-b948-1d614b932fd7","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:07:39Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"d86db5e3-6a6a-4baf-b948-1d614b932fd7","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:08:11Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"b7b7a351-38ec-40c1-9df4-3024c90ae594"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:08:11Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"b7b7a351-38ec-40c1-9df4-3024c90ae594","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:08:11Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"b7b7a351-38ec-40c1-9df4-3024c90ae594","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:09:15Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"1c713bc0-e06b-4e24-b960-da11bd661ccf"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:09:15Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"1c713bc0-e06b-4e24-b960-da11bd661ccf","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:09:15Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"1c713bc0-e06b-4e24-b960-da11bd661ccf","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:11:15Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"f1ab4ccc-918b-4113-9d68-0590b91bbd7a"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:11:15Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"f1ab4ccc-918b-4113-9d68-0590b91bbd7a","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:11:15Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"f1ab4ccc-918b-4113-9d68-0590b91bbd7a","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:13:15Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"fd892d95-323b-4a74-a29d-e34576713c23"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:15Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"fd892d95-323b-4a74-a29d-e34576713c23","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:15Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"fd892d95-323b-4a74-a29d-e34576713c23","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:13:20Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"1f1dc006-90f4-44f6-a60d-7d8bca23744d"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:20Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"1f1dc006-90f4-44f6-a60d-7d8bca23744d","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:20Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"1f1dc006-90f4-44f6-a60d-7d8bca23744d","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:13:20Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"f02a27ec-9ff7-4e26-948d-5e83296618bc"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:20Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"f02a27ec-9ff7-4e26-948d-5e83296618bc","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:20Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"f02a27ec-9ff7-4e26-948d-5e83296618bc","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:13:24Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"1c1d2145-a71b-49f5-aacd-7e80abf860e4"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:24Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"1c1d2145-a71b-49f5-aacd-7e80abf860e4","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:24Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"1c1d2145-a71b-49f5-aacd-7e80abf860e4","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:13:32Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"6b7361a7-0aa3-45a4-85ef-f8758103fe20"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:32Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"6b7361a7-0aa3-45a4-85ef-f8758103fe20","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:32Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"6b7361a7-0aa3-45a4-85ef-f8758103fe20","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:13:48Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"d494126e-d245-4658-829a-a8faf6793ba7"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:48Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"d494126e-d245-4658-829a-a8faf6793ba7","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:13:48Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"d494126e-d245-4658-829a-a8faf6793ba7","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:14:20Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"6a86e24a-cf09-4a97-a691-9648bf78ea56"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:14:20Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"6a86e24a-cf09-4a97-a691-9648bf78ea56","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:14:20Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"6a86e24a-cf09-4a97-a691-9648bf78ea56","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:15:25Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"e66399e8-4ca3-4284-b2d8-38f62372892c"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:15:25Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"e66399e8-4ca3-4284-b2d8-38f62372892c","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:15:25Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"e66399e8-4ca3-4284-b2d8-38f62372892c","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"debug","ts":"2026-06-15T06:17:25Z","caller":"handlers/server.go:214","msg":"EPP received request","x-request-id":"c1345da8-5aef-459b-982b-12b5bb953da9"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:17:25Z","caller":"handlers/server.go:235","msg":"Error handling request","x-request-id":"c1345da8-5aef-459b-982b-12b5bb953da9","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:235\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] {"level":"error","ts":"2026-06-15T06:17:25Z","caller":"handlers/server.go:322","msg":"Failed to process request","x-request-id":"c1345da8-5aef-459b-982b-12b5bb953da9","error":"inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request","stacktrace":"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process\n\t/go/pkg/mod/sigs.k8s.io/gateway-api-inference-extension@v1.4.0/pkg/epp/handlers/server.go:322\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1715\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1859\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.79.3/server.go:1064"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'tokenizer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 INFO 06-15 06:03:19 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 06-15 06:03:19 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] 2026-06-15 06:03:21,407 [INFO] [root] TokenizationServiceServicer initialized [e2e-llm-inference-service] 2026-06-15 06:03:21,408 [INFO] [root] gRPC reflection disabled (set `ENABLE_GRPC_REFLECTION=1` to enable) [e2e-llm-inference-service] 2026-06-15 06:03:21,408 [INFO] [root] gRPC server configured to listen on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:03:21,408 [INFO] [root] gRPC server started on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:03:21,409 [INFO] [root] Probe server started on port 8082 [e2e-llm-inference-service] 2026-06-15 06:03:21,409 [INFO] [root] Server started. [e2e-llm-inference-service] 2026-06-15 06:03:22,049 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:03:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:03:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:03:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:03:27,046 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:03:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:03:32,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:03:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:03:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:03:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:03:42,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:03:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:03:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:03:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:03:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:03:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:04:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:04:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:12,046 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:12,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:27 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:42,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:05:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:05:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:22,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:32 +0000] "GET /healthz HTTP/1.1" 200 261 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:06:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:12,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:32,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:42,048 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:52,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:07:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:02,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:22,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:08:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:02,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:22,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:32,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:42,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:09:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:12,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:10:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:02,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:22,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:32,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:52,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:52 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:11:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:02,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:52,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:52 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:12:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:12,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:42,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:13:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:02,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:32,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:14:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:42,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:15:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:32,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:42,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:52 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:16:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:02,965 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:12,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:12,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:22,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:27,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:32,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:32 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:42,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:42,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:52,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:57,047 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:17:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:02,964 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:18:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 821ed6f2-84c6-47d6-ab6d-5fb4168751a9 [e2e-llm-inference-service] resourceVersion: '24855' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:03:43Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.31 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] uid: 9359d817-81c6-4fe2-80e0-76d36607616d [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: bdbda0a8-b4c3-40ae-96b9-9090afc7ca02 [e2e-llm-inference-service] resourceVersion: '24349' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - notReadyAddresses: [e2e-llm-inference-service] - ip: 10.133.0.35 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] uid: 03be4983-7e0f-4763-b9f9-537fa20b4610 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] generateName: auth-enabled-test-kserve-85d86d876c- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 03be4983-7e0f-4763-b9f9-537fa20b4610 [e2e-llm-inference-service] resourceVersion: '24347' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85d86d876c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.35/23"],"mac_address":"0a:58:0a:85:00:23","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.35/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.35\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:23\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: auth-enabled-test-kserve-85d86d876c [e2e-llm-inference-service] uid: afbab033-b48f-4827-a683-4e1cc3932d27 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-25 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"afbab033-b48f-4827-a683-4e1cc3932d27"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.35"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-d2vjn [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-d2vjn [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n ucx_hcas=()\n for hca in \"${active_hcas[@]}\"\ [e2e-llm-inference-service] ; do\n ucx_hcas+=(\"${hca}:1\")\n done\n\n # Check if we found any active\ [e2e-llm-inference-service] \ HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements\ [e2e-llm-inference-service] \ with a comma\n hcas=$(IFS=,; echo \"${active_hcas[*]}\")\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Setting active HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n\ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if counts are equal - prefer lower index number\n\ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8000\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-d2vjn [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:16Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [storage-initializer]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] hostIP: 10.0.141.25 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.25 [e2e-llm-inference-service] podIP: 10.133.0.35 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.35 [e2e-llm-inference-service] startTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:03:15Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://8b4c36867ef169438602cb7c2bd89a577c73c2c0994c64f9ca6b693147449951 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-d2vjn [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-d2vjn [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] generateName: auth-enabled-test-kserve-router-scheduler-6c5d597fbb- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9359d817-81c6-4fe2-80e0-76d36607616d [e2e-llm-inference-service] resourceVersion: '24854' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6c5d597fbb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.31/23"],"mac_address":"0a:58:0a:86:00:1f","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.31/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.31\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:1f\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-6c5d597fbb [e2e-llm-inference-service] uid: eedb0e24-1784-41de-852e-01b08abb9f57 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-226 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"eedb0e24-1784-41de-852e-01b08abb9f57"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.31"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-h88n7 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-h88n7 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-enabled-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-h88n7 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] - name: kube-api-access-h88n7 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-enabled-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-enabled-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: auth-enabled-test-epp-sa-dockercfg-dz9xz [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:13Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:43Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:43Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] hostIP: 10.0.128.226 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.226 [e2e-llm-inference-service] podIP: 10.134.0.31 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.31 [e2e-llm-inference-service] startTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] containerID: cri-o://55bcaeaed2a108002a4cc5fd3a43c9f6c911f9361281b7d4e5a71122e0d9f91c [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://55bcaeaed2a108002a4cc5fd3a43c9f6c911f9361281b7d4e5a71122e0d9f91c [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-h88n7 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:03:13Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-scheduler@sha256:88de279c6eb6758a4c600de9730e49e46b04c392846afedd03d82447379c9e7a [e2e-llm-inference-service] containerID: cri-o://f17a74289ac80e1118204b380d94d45d17e4d27afd28e9cb9557783dbe08aeef [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-h88n7 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:03:14Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-uds-tokenizer@sha256:aed091a51f3d64458f1fdb451d21f745186bb4517a7ba0c49913a0c617366a3e [e2e-llm-inference-service] containerID: cri-o://d98757a96b0a8e28ff609c63ebd79b7448768eeb80b847f7b2eaf808789a37ce [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-h88n7 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1012b5c9-0e08-40ff-84be-dbca743619c9 [e2e-llm-inference-service] resourceVersion: '24182' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: auth-enabled-test-epp-sa-dockercfg-dz9xz [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"auth-enabled-test-epp-sa-dockercfg-dz9xz"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjfwp"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: auth-enabled-test-epp-sa-dockercfg-dz9xz [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: auth-enabled-test-epp-sa-dockercfg-dz9xz [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e0dd11a3-22ce-433a-8570-b647c66d97bb [e2e-llm-inference-service] resourceVersion: '24204' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.49.74 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.49.74 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a31b13b7-b9e3-4ceb-90c0-d2648f555a09 [e2e-llm-inference-service] resourceVersion: '24172' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.166.190 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.166.190 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: bab81096-d0c8-40e1-bbc4-635e04522112 [e2e-llm-inference-service] resourceVersion: '31661' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:13:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:13:12Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:13:12Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "auth-enabled-test-kserve-85d86d876c" has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 21e6fa54-6782-4056-8977-4ca5d2d316f9 [e2e-llm-inference-service] resourceVersion: '24858' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-enabled-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-enabled-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-enabled-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:03:44Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:44Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:03:44Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "auth-enabled-test-kserve-router-scheduler-6c5d597fbb" has [e2e-llm-inference-service] successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-85d86d876c [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: afbab033-b48f-4827-a683-4e1cc3932d27 [e2e-llm-inference-service] resourceVersion: '24189' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85d86d876c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: auth-enabled-test-kserve [e2e-llm-inference-service] uid: bab81096-d0c8-40e1-bbc4-635e04522112 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"bab81096-d0c8-40e1-bbc4-635e04522112"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85d86d876c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85d86d876c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-6c5d597fbb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: eedb0e24-1784-41de-852e-01b08abb9f57 [e2e-llm-inference-service] resourceVersion: '24857' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6c5d597fbb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 21e6fa54-6782-4056-8977-4ca5d2d316f9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"21e6fa54-6782-4056-8977-4ca5d2d316f9"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6c5d597fbb [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6c5d597fbb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-enabled-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-enabled-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-enabled-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e5de7ed8-1ffd-4a45-9689-77502e2b8662 [e2e-llm-inference-service] resourceVersion: '24197' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: auth-enabled-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: auth-enabled-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e0239804-896e-4a33-b3dd-1ce2cbe4d811 [e2e-llm-inference-service] resourceVersion: '24195' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-service-xlvf4 [e2e-llm-inference-service] generateName: auth-enabled-test-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6ec5c79e-ca71-496f-a62d-2ad94a344b26 [e2e-llm-inference-service] resourceVersion: '24856' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: auth-enabled-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:03:43Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] uid: e0dd11a3-22ce-433a-8570-b647c66d97bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e0dd11a3-22ce-433a-8570-b647c66d97bb"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.31 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] uid: 9359d817-81c6-4fe2-80e0-76d36607616d [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc-v5v46 [e2e-llm-inference-service] generateName: auth-enabled-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 48ab8eb9-b0aa-4894-8fbe-4f1f2124e1e2 [e2e-llm-inference-service] resourceVersion: '24348' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] uid: a31b13b7-b9e3-4ceb-90c0-d2648f555a09 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a31b13b7-b9e3-4ceb-90c0-d2648f555a09"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.35 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] serving: false [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] uid: 03be4983-7e0f-4763-b9f9-537fa20b4610 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e5de7ed8-1ffd-4a45-9689-77502e2b8662 [e2e-llm-inference-service] resourceVersion: '24197' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:auth-enabled-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: auth-enabled-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: auth-enabled-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e0239804-896e-4a33-b3dd-1ce2cbe4d811 [e2e-llm-inference-service] resourceVersion: '24195' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24378' [e2e-llm-inference-service] uid: bd3b245e-8a97-4a60-a0ce-375438effc24 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/auth-enabled-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/auth-enabled-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/auth-enabled-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/auth-enabled-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24378' [e2e-llm-inference-service] uid: bd3b245e-8a97-4a60-a0ce-375438effc24 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/auth-enabled-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/auth-enabled-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/auth-enabled-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/auth-enabled-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24363' [e2e-llm-inference-service] uid: 388a36fb-c476-41a0-a378-228288126770 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24247' [e2e-llm-inference-service] uid: cc35d952-6ecc-4ca3-9917-8478d490f85f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24372' [e2e-llm-inference-service] uid: 705bdb6a-b9c8-40cf-90ee-49dd2649a4f9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-inference-pool-ip-0e6361b2.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24276' [e2e-llm-inference-service] uid: 6bc6f69b-633b-4579-9507-abe992fa9bf9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24247' [e2e-llm-inference-service] uid: cc35d952-6ecc-4ca3-9917-8478d490f85f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24372' [e2e-llm-inference-service] uid: 705bdb6a-b9c8-40cf-90ee-49dd2649a4f9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-inference-pool-ip-0e6361b2.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24276' [e2e-llm-inference-service] uid: 6bc6f69b-633b-4579-9507-abe992fa9bf9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24247' [e2e-llm-inference-service] uid: cc35d952-6ecc-4ca3-9917-8478d490f85f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:17Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24372' [e2e-llm-inference-service] uid: 705bdb6a-b9c8-40cf-90ee-49dd2649a4f9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-inference-pool-ip-0e6361b2.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:12Z' [e2e-llm-inference-service] name: auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24276' [e2e-llm-inference-service] uid: 6bc6f69b-633b-4579-9507-abe992fa9bf9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-enabled-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"313eeec2-288b-40e6-a85a-a6d15c6bb352"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:03:11Z' [e2e-llm-inference-service] name: auth-enabled-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-enabled-test [e2e-llm-inference-service] uid: 313eeec2-288b-40e6-a85a-a6d15c6bb352 [e2e-llm-inference-service] resourceVersion: '24213' [e2e-llm-inference-service] uid: 12bb4b0e-34bd-43a7-903f-fa9c1ed4abaf [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-enabled-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:18:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-enabled-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6c5d597fbb [e2e-llm-inference-service] timestamp: '2026-06-15T06:17:52Z' [e2e-llm-inference-service] window: 14.923s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 245198n [e2e-llm-inference-service] memory: 359880Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 555317n [e2e-llm-inference-service] memory: 21600Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_auth.py:374 ⏭️ Skipping deletion of auth-enabled-test due to test failure (SKIP_DELETION_ON_FAILURE=True) [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_auth_enabled_requires_token] [2026-06-15T06:18:10.621845] end - ❌ 902.127s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:03:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:03:17Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-llmd-simulator1] ______ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-router-m... {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:727: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'workload-llm... {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-06-15T06:07:11.714988', start_time = 1781503631.7154357 [e2e-llm-inference-service] duration = 1102.550819158554, timestamp_end = '2026-06-15T06:25:34.266259' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-router-m... {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] extra_headers = {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'} [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1030: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f7425e9a020> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1026: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-managed-llmisvc-router-m-0cc5ee6c in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-router-m-0cc5ee6c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-router-m-0cc5ee6c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-8461fd55 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-8461fd55 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-8461fd55 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-qw-ab631bc0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-qw-ab631bc0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-qw-ab631bc0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-ef361761 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-ef361761 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-ef361761 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-5aed586c in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-5aed586c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-5aed586c [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-06-15T06:06:31.439660] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-router-managed-test-llm-5b1e8f15', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2fce0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[TestCase(base_refs=['router-managed', 'workload-llmd-simulator', 'model-qwen2.5-0.5b'], prompt='KServe is a', service_name='llmisvc-model-qwen2-5-0-5b-rout-a50492e9', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2fd80>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/Qwen/Qwen2.5-0.5B-Instruct'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-qwen2-5-0-5b-rout-a50492e9', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-qw-ab631bc0'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-ef361761'}, [e2e-llm-inference-service] {'name': 'model-qwen2-5-0-5b-llmisvc-mode-5aed586c'}]}, [e2e-llm-inference-service] 'status': None}, model_name='Qwen/Qwen2.5-0.5B-Instruct')], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-06-15T06:06:31.460238] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-06-15T06:06:31.516172] end - ✅ in 0.056s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-06-15T06:06:31.516280] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:06:40Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:06:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:06:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:06:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:06:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:06:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:06:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:06:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:06:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:06:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:06:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:06:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:06:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:06:49Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-06-15T06:07:11.714822] end - ✅ in 40.198s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-06-15T06:07:11.714988] start - args=(, TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-router-managed-test-llm-5b1e8f15', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2fce0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[TestCase(base_refs=['router-managed', 'workload-llmd-simulator', 'model-qwen2.5-0.5b'], prompt='KServe is a', service_name='llmisvc-model-qwen2-5-0-5b-rout-a50492e9', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2fd80>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/Qwen/Qwen2.5-0.5B-Instruct'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-qwen2-5-0-5b-rout-a50492e9', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-qw-ab631bc0'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-ef361761'}, [e2e-llm-inference-service] {'name': 'model-qwen2-5-0-5b-llmisvc-mode-5aed586c'}]}, [e2e-llm-inference-service] 'status': None}, model_name='Qwen/Qwen2.5-0.5B-Instruct')], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T06:07:11.715458] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T06:07:11.723745] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T06:13:20.880487] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T06:13:20.919252] end - ✅ in 0.038s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T06:19:30.068349] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T06:19:30.107512] end - ✅ in 0.038s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1130 Timed out waiting: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-06-15T06:25:34.266259] end - ❌ 1102.551s: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:742 [router-managed-workload-llmd-simulator] ❌ ERROR: Failed to call llm inference service llmisvc-router-managed-test-llm-5b1e8f15: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1151 🔍 # Diagnostics for 'llmisvc-router-managed-test-llm-5b1e8f15' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1152 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1153 # LLMInferenceService llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1156 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:31Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:31Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:31Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '27859' [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-router-m-0cc5ee6c [e2e-llm-inference-service] - name: workload-llmd-simulator-llmisvc-8461fd55 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:48Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:48Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:40Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-699694bb49-m6gc4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-b5799d8f5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-699694bb49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-85d86d876c-vrqhw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" in 3.371s (3.371s including waiting). Image size: 299992506 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6c5d597fbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-85d86d876c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.36s (27.36s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-f5744d7b7-gjb94 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-7748b48dbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-f5744d7b7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-779977f94c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.286s (1.286s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:50 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:35 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:20 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:07:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7bc88f48bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-548bd48954 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-67h82 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.023s (1.023s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-67h82 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-h6wcn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.38:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-74dcd66d7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5c556785f6 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.707s (2.707s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:33 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-75dcfd69c9-dh6qf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-75dcfd69c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:44 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-96f8b89cb-j7r99 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-96f8b89cb-j7r99 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-9c4c7855f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-96f8b89cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.082s (1.082s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 951ms (951ms including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.592s (30.592s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.034s (1.034s including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 31.996s (31.996s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.34:8082/healthz": dial tcp 10.133.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884fbb from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5d7479f884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 I0615 06:06:39.671497 1 config.go:602] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": 100000000, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "inter-token-latency": 0, [e2e-llm-inference-service] "inter-token-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": 0, [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": 0, [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "port": 8000, [e2e-llm-inference-service] "prefill-overhead": 0, [e2e-llm-inference-service] "prefill-time-per-token": 0, [e2e-llm-inference-service] "prefill-time-std-dev": 0, [e2e-llm-inference-service] "seed": 1781503599671030800, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": 0, [e2e-llm-inference-service] "time-to-first-token-std-dev": 0, [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket", [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0615 06:06:39.705382 1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0615 06:06:39.710600 1 context.go:138] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0615 06:06:39.710659 1 communication.go:49] "Starting communication layer" [e2e-llm-inference-service] I0615 06:06:39.710719 1 simulator.go:188] "Start processing routine" [e2e-llm-inference-service] I0615 06:06:39.710947 1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0615 06:06:39.710972 1 grpc.go:126] "Server starting" protocol="gRPC" port=8000 [e2e-llm-inference-service] I0615 06:06:39.711633 1 http.go:96] "Server starting" protocol="HTTPS" port=8000 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"setup","caller":"runner/runner.go:150","msg":"GIE build","commit-sha":"","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"setup","caller":"runner/runner.go:169","msg":"Flags processed","flags":{"cache-info-metric":"vllm:cache_config_info","cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"kv-cache-usage-percentage-metric":"vllm:kv_cache_usage_perc","lora-info-metric":"vllm:lora_requests_info","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"total-queued-requests-metric":"vllm:num_requests_waiting","total-running-requests-metric":"vllm:num_requests_running","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"setup.trace","caller":"tracing/telemetry.go:131","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"loader/configloader.go:65","msg":"Loaded raw configuration","config":"{FeatureGates: {}, Plugins: [{/single-profile-handler} {/queue-scorer} {/prefix-cache-scorer} {/max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","caller":"prefix/plugin.go:203","msg":"BlockSize is not positive, using default value","default":16} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","caller":"prefix/plugin.go:213","msg":"PrefixCachePlugin initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":256,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"loader/configloader.go:98","msg":"Effective configuration loaded","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","kind":"EndpointPickerConfig"},"configError":"got runtime.Object without object metadata: {FeatureGates: {}, Plugins: [{single-profile-handler/single-profile-handler} {queue-scorer/queue-scorer} {prefix-cache-scorer/prefix-cache-scorer} {max-score-picker/max-score-picker} {fcfs-ordering-policy/fcfs-ordering-policy} {global-strict-fairness-policy/global-strict-fairness-policy}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"runner/runner.go:549","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"setup","caller":"runner/runner.go:301","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"setup","caller":"runner/runner.go:315","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","logger":"setup.SaturationDetector","caller":"utilizationdetector/detector.go:70","msg":"Creating new SaturationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"setup","caller":"runner/runner.go:350","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"setup","caller":"runner/runner.go:644","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"setup","caller":"runner/runner.go:209","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:39Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:39Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:40Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:40Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:40Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:40Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:40Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:40Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:40Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","reconcileID":"6abc2002-75de-4d14-aded-41cc4be45079","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:40Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:06:40Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:47Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","reconcileID":"33e13048-1485-452b-bea4-fd0d69a0e665","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:49Z","caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8","reconcileID":"e3803c47-62d4-4cb3-a57a-8aa2ac617cf3"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:06:49Z","caller":"metrics/pod_metrics.go:76","msg":"Starting refresher","endpoint":{"name":"llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8-rank-0","namespace":"kserve-ci-e2e-test"},"metadata":"{NamespacedName:kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8-rank-0 PodName:llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 Address:10.132.0.45 Port:8000 MetricsHost:10.132.0.45:8000 Labels:map[app.kubernetes.io/component:llminferenceservice-workload app.kubernetes.io/name:llmisvc-router-managed-test-llm-5b1e8f15 app.kubernetes.io/part-of:llminferenceservice kserve.io/component:workload llm-d.ai/role:both pod-template-hash:7c5bd57d44]}"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'tokenizer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 INFO 06-15 06:06:45 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 06-15 06:06:45 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] 2026-06-15 06:06:47,121 [INFO] [root] TokenizationServiceServicer initialized [e2e-llm-inference-service] 2026-06-15 06:06:47,122 [INFO] [root] gRPC reflection disabled (set `ENABLE_GRPC_REFLECTION=1` to enable) [e2e-llm-inference-service] 2026-06-15 06:06:47,122 [INFO] [root] gRPC server configured to listen on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:06:47,122 [INFO] [root] gRPC server started on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:06:47,122 [INFO] [root] Probe server started on port 8082 [e2e-llm-inference-service] 2026-06-15 06:06:47,123 [INFO] [root] Server started. [e2e-llm-inference-service] 2026-06-15 06:06:49,685 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:06:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:06:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:06:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:06:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:09,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:09 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:20 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:07:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:07:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:00,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:20 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:40,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:08:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:08:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:10,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:20,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:20 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:39 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:40,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:40 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:09:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:09:54 +0000] "GET /healthz HTTP/1.1" 200 261 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:20,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:20 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:40 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:50 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:10:54,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:10:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:00 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:20,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:40,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:11:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:11:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:09,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:09 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:10,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:39,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:12:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:12:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:09,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:10 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:20,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:13:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:13:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:20,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:24,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:14:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:14:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:00,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:10,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:40 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:50 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:15:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:15:54 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:00,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:10,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:20 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:30,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:50,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:50 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:16:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:16:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:00 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:09,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:30,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:39 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:40 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:50,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:17:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:17:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:00,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:00 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:10,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:20,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:39,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:50,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:18:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:18:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:19:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:19:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:20,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:20 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:20:54,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:20:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:24,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:30,176 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:21:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:21:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:09,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:10 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:50 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:22:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:22:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:20 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:39,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:39 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:23:54,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:23:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:09 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:39,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:40,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:50,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:50 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:24:54,684 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:24:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:25:00,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:25:09,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:09 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:25:10,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:25:20,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:25:24,683 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:25:30,177 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b93a0882-0e6a-4652-aec1-dafe865bb9aa [e2e-llm-inference-service] resourceVersion: '27854' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.40 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] uid: 1264e5f9-09ea-4712-8848-1be6c22009fa [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: dfc823bf-0610-430c-a700-d8548a3450e6 [e2e-llm-inference-service] resourceVersion: '27610' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.45 [e2e-llm-inference-service] nodeName: ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] uid: be8770ba-8db8-4a0c-99b7-a16c4f499392 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: be8770ba-8db8-4a0c-99b7-a16c4f499392 [e2e-llm-inference-service] resourceVersion: '27608' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 7c5bd57d44 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.45/23"],"mac_address":"0a:58:0a:84:00:2d","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.45/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.45\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:2d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 [e2e-llm-inference-service] uid: b6522ee7-6b00-47fc-8f8e-7b5f486604f6 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-243 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b6522ee7-6b00-47fc-8f8e-7b5f486604f6"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.45"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-82nwf [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-82nwf [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:40Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] hostIP: 10.0.128.243 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.243 [e2e-llm-inference-service] podIP: 10.132.0.45 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.45 [e2e-llm-inference-service] startTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim@sha256:bab162bd25e2ed8b15022387cdb223023aeb33be49476af9f0115c0398fb8ff5 [e2e-llm-inference-service] containerID: cri-o://cfc86b1647130920b0fede877a2100ac2a0ef189a788249428d22f85aa834abd [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-82nwf [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] generateName: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1264e5f9-09ea-4712-8848-1be6c22009fa [e2e-llm-inference-service] resourceVersion: '27852' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 68b6785c7d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.40/23"],"mac_address":"0a:58:0a:85:00:28","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.40/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.40\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:28\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d [e2e-llm-inference-service] uid: 0a54deb4-75ba-4b8b-8fb7-4a4fe96d4f69 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-25 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0a54deb4-75ba-4b8b-8fb7-4a4fe96d4f69"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.40"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-7jg2d [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-7jg2d [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-7jg2d [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-rx77b [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:40Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] hostIP: 10.0.141.25 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.25 [e2e-llm-inference-service] podIP: 10.133.0.40 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.40 [e2e-llm-inference-service] startTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-scheduler@sha256:88de279c6eb6758a4c600de9730e49e46b04c392846afedd03d82447379c9e7a [e2e-llm-inference-service] containerID: cri-o://c2d6e31fd4caa01e365ddd5e74ad16e61669ffc6a74474a0486fcc133a9a87b9 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-7jg2d [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-uds-tokenizer@sha256:aed091a51f3d64458f1fdb451d21f745186bb4517a7ba0c49913a0c617366a3e [e2e-llm-inference-service] containerID: cri-o://f57a309a3b8c5c8eee249eff5a705a9ebacae81579281f80635f6407af08c507 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-7jg2d [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ac1ce945-7d0a-4527-9122-d81598e8c8aa [e2e-llm-inference-service] resourceVersion: '27360' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-rx77b [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-rx77b"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjfwp"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-rx77b [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-rx77b [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d5700ec4-b286-4524-9c5a-ea2e9d9d3df1 [e2e-llm-inference-service] resourceVersion: '27391' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.233.24 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.233.24 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a9d77445-d080-445b-8809-55479ab1b123 [e2e-llm-inference-service] resourceVersion: '27349' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.5.69 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.5.69 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d9d19e2e-683b-47fc-916b-45c9bc224baa [e2e-llm-inference-service] resourceVersion: '27614' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1692e624-f633-477f-917e-ebb5a2755d26 [e2e-llm-inference-service] resourceVersion: '27856' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b6522ee7-6b00-47fc-8f8e-7b5f486604f6 [e2e-llm-inference-service] resourceVersion: '27613' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 7c5bd57d44 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] uid: d9d19e2e-683b-47fc-916b-45c9bc224baa [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d9d19e2e-683b-47fc-916b-45c9bc224baa"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 7c5bd57d44 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 7c5bd57d44 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0a54deb4-75ba-4b8b-8fb7-4a4fe96d4f69 [e2e-llm-inference-service] resourceVersion: '27855' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 68b6785c7d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] uid: 1692e624-f633-477f-917e-ebb5a2755d26 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1692e624-f633-477f-917e-ebb5a2755d26"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 68b6785c7d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 68b6785c7d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9df477c1-23f8-4449-80b2-26c9cfd17a27 [e2e-llm-inference-service] resourceVersion: '27373' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 27f4d143-6011-40be-8f99-81dee5eaaf27 [e2e-llm-inference-service] resourceVersion: '27370' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service-h9pwm [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-5b1e8f15-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a6dc8878-f6c5-44d7-81b2-9fe391d437a0 [e2e-llm-inference-service] resourceVersion: '27853' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] uid: d5700ec4-b286-4524-9c5a-ea2e9d9d3df1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:07:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5700ec4-b286-4524-9c5a-ea2e9d9d3df1"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.40 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] uid: 1264e5f9-09ea-4712-8848-1be6c22009fa [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-sg8ftz [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c5eb603c-cd68-477a-9d8f-8c63643b27e0 [e2e-llm-inference-service] resourceVersion: '27611' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] uid: a9d77445-d080-445b-8809-55479ab1b123 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a9d77445-d080-445b-8809-55479ab1b123"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.45 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] uid: be8770ba-8db8-4a0c-99b7-a16c4f499392 [e2e-llm-inference-service] nodeName: ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9df477c1-23f8-4449-80b2-26c9cfd17a27 [e2e-llm-inference-service] resourceVersion: '27373' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 27f4d143-6011-40be-8f99-81dee5eaaf27 [e2e-llm-inference-service] resourceVersion: '27370' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27584' [e2e-llm-inference-service] uid: e979a886-d13c-4845-82cd-893e7278daac [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27584' [e2e-llm-inference-service] uid: e979a886-d13c-4845-82cd-893e7278daac [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27573' [e2e-llm-inference-service] uid: c1869303-4862-4b42-a58f-79748b4ddedf [e2e-llm-inference-service] spec: [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:41Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:41Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:06:41Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27514' [e2e-llm-inference-service] uid: 76305db5-4f54-4ac9-a8a0-5b5828ae538e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:41Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:06:41Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27407' [e2e-llm-inference-service] uid: 3ed7fa3d-ca61-49ff-9c8a-484746c8662a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27578' [e2e-llm-inference-service] uid: f9850cd5-7b1e-43f3-b870-f52fb13858f8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-inference--ip-33d4275d.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27426' [e2e-llm-inference-service] uid: ac0ce4b7-f6a2-4e0b-8052-58055a87f1d7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27407' [e2e-llm-inference-service] uid: 3ed7fa3d-ca61-49ff-9c8a-484746c8662a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27578' [e2e-llm-inference-service] uid: f9850cd5-7b1e-43f3-b870-f52fb13858f8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-inference--ip-33d4275d.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27426' [e2e-llm-inference-service] uid: ac0ce4b7-f6a2-4e0b-8052-58055a87f1d7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27407' [e2e-llm-inference-service] uid: 3ed7fa3d-ca61-49ff-9c8a-484746c8662a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:47Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27578' [e2e-llm-inference-service] uid: f9850cd5-7b1e-43f3-b870-f52fb13858f8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-inference--ip-33d4275d.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27426' [e2e-llm-inference-service] uid: ac0ce4b7-f6a2-4e0b-8052-58055a87f1d7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1eb14ae-2de1-45da-9dac-2695ca3de07a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:06:39Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: d1eb14ae-2de1-45da-9dac-2695ca3de07a [e2e-llm-inference-service] resourceVersion: '27398' [e2e-llm-inference-service] uid: 76a32537-92ed-4bdd-b6f7-61f09224e7d6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:35Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 7c5bd57d44 [e2e-llm-inference-service] timestamp: '2026-06-15T06:25:17Z' [e2e-llm-inference-service] window: 12.37s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 8444219n [e2e-llm-inference-service] memory: 25616Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:35Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 68b6785c7d [e2e-llm-inference-service] timestamp: '2026-06-15T06:25:11Z' [e2e-llm-inference-service] window: 15.626s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 10391890n [e2e-llm-inference-service] memory: 25628Ki [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 267182n [e2e-llm-inference-service] memory: 359924Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-06-15T06:25:35.816379] end - ❌ 1144.376s: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-llmd-simulator2] ______ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='What is KServe?', service_name='llmisvc-rout... {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:727: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'workload-llm... {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-06-15T06:26:15.484944', start_time = 1781504775.4854898 [e2e-llm-inference-service] duration = 1102.5697031021118, timestamp_end = '2026-06-15T06:44:38.055197' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='What is KServe?', service_name='llmisvc-rout... {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] extra_headers = {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'} [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1030: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f7425e99760> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1026: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-managed-llmisvc-router-m-d2b356d9 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-router-m-d2b356d9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-router-m-d2b356d9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-53a6ad30 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-53a6ad30 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-53a6ad30 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-qw-a8136937 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-qw-a8136937 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-qw-a8136937 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-54bae102 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-54bae102 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-54bae102 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-21bdc061 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-21bdc061 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-21bdc061 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-06-15T06:25:36.155635] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='What is KServe?', service_name='llmisvc-router-managed-test-llm-4b931143', endpoint='/v1/chat/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2fe20>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[TestCase(base_refs=['router-managed', 'workload-llmd-simulator', 'model-qwen2.5-0.5b'], prompt='What is KServe?', service_name='llmisvc-model-qwen2-5-0-5b-rout-4f8c0978', endpoint='/v1/chat/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2fec0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/Qwen/Qwen2.5-0.5B-Instruct'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-qwen2-5-0-5b-rout-4f8c0978', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-qw-a8136937'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-54bae102'}, [e2e-llm-inference-service] {'name': 'model-qwen2-5-0-5b-llmisvc-mode-21bdc061'}]}, [e2e-llm-inference-service] 'status': None}, model_name='Qwen/Qwen2.5-0.5B-Instruct')], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-06-15T06:25:36.168327] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-06-15T06:25:36.213617] end - ✅ in 0.045s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-06-15T06:25:36.213875] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:25:44Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:25:44Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:25:44Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:25:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:25:44Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:25:44Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:25:44Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:25:44Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:25:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:25:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:25:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:25:44Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:25:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:25:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:25:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:25:55Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-06-15T06:26:15.484462] end - ✅ in 39.270s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-06-15T06:26:15.484944] start - args=(, TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='What is KServe?', service_name='llmisvc-router-managed-test-llm-4b931143', endpoint='/v1/chat/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2fe20>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[TestCase(base_refs=['router-managed', 'workload-llmd-simulator', 'model-qwen2.5-0.5b'], prompt='What is KServe?', service_name='llmisvc-model-qwen2-5-0-5b-rout-4f8c0978', endpoint='/v1/chat/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2fec0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/Qwen/Qwen2.5-0.5B-Instruct'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-qwen2-5-0-5b-rout-4f8c0978', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-qw-a8136937'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-54bae102'}, [e2e-llm-inference-service] {'name': 'model-qwen2-5-0-5b-llmisvc-mode-21bdc061'}]}, [e2e-llm-inference-service] 'status': None}, model_name='Qwen/Qwen2.5-0.5B-Instruct')], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T06:26:15.485508] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T06:26:15.493860] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T06:32:24.658550] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T06:32:24.701040] end - ✅ in 0.042s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T06:38:33.847855] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T06:38:33.892214] end - ✅ in 0.044s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1130 Timed out waiting: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-06-15T06:44:38.055197] end - ❌ 1102.570s: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:742 [router-managed-workload-llmd-simulator] ❌ ERROR: Failed to call llm inference service llmisvc-router-managed-test-llm-4b931143: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1151 🔍 # Diagnostics for 'llmisvc-router-managed-test-llm-4b931143' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1152 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1153 # LLMInferenceService llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1156 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:36Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:36Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:36Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '41341' [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-router-m-d2b356d9 [e2e-llm-inference-service] - name: workload-llmd-simulator-llmisvc-53a6ad30 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:55Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:55Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:55Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:44Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:55Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-699694bb49-m6gc4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-b5799d8f5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-699694bb49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-85d86d876c-vrqhw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" in 3.371s (3.371s including waiting). Image size: 299992506 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6c5d597fbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-85d86d876c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.36s (27.36s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-f5744d7b7-gjb94 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-7748b48dbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-f5744d7b7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:25 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-598d8c75cc-qw9md [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-54bd696fdf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-598d8c75cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-779977f94c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.286s (1.286s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:50 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:07 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:35 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:20 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:07:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7bc88f48bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-548bd48954 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-67h82 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.023s (1.023s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-67h82 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-h6wcn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.38:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-74dcd66d7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5c556785f6 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.707s (2.707s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:33 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-75dcfd69c9-dh6qf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-75dcfd69c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:59 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:32:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7d4868d689 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-578d595fc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-96f8b89cb-j7r99 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-96f8b89cb-j7r99 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-9c4c7855f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-96f8b89cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.082s (1.082s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 951ms (951ms including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.592s (30.592s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.034s (1.034s including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 31.996s (31.996s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.34:8082/healthz": dial tcp 10.133.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884fbb from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5d7479f884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 I0615 06:25:42.291637 1 config.go:602] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": 100000000, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "inter-token-latency": 0, [e2e-llm-inference-service] "inter-token-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": 0, [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": 0, [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "port": 8000, [e2e-llm-inference-service] "prefill-overhead": 0, [e2e-llm-inference-service] "prefill-time-per-token": 0, [e2e-llm-inference-service] "prefill-time-std-dev": 0, [e2e-llm-inference-service] "seed": 1781504742291157800, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": 0, [e2e-llm-inference-service] "time-to-first-token-std-dev": 0, [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket", [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0615 06:25:42.326702 1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0615 06:25:42.331451 1 context.go:138] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0615 06:25:42.331515 1 communication.go:49] "Starting communication layer" [e2e-llm-inference-service] I0615 06:25:42.331533 1 simulator.go:188] "Start processing routine" [e2e-llm-inference-service] I0615 06:25:42.331774 1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0615 06:25:42.331812 1 grpc.go:126] "Server starting" protocol="gRPC" port=8000 [e2e-llm-inference-service] I0615 06:25:42.332480 1 http.go:96] "Server starting" protocol="HTTPS" port=8000 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"setup","caller":"runner/runner.go:150","msg":"GIE build","commit-sha":"","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"setup","caller":"runner/runner.go:169","msg":"Flags processed","flags":{"cache-info-metric":"vllm:cache_config_info","cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"kv-cache-usage-percentage-metric":"vllm:kv_cache_usage_perc","lora-info-metric":"vllm:lora_requests_info","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"total-queued-requests-metric":"vllm:num_requests_waiting","total-running-requests-metric":"vllm:num_requests_running","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"setup.trace","caller":"tracing/telemetry.go:131","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"loader/configloader.go:65","msg":"Loaded raw configuration","config":"{FeatureGates: {}, Plugins: [{/single-profile-handler} {/queue-scorer} {/prefix-cache-scorer} {/max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"prefix/plugin.go:203","msg":"BlockSize is not positive, using default value","default":16} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"prefix/plugin.go:213","msg":"PrefixCachePlugin initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":256,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"loader/configloader.go:98","msg":"Effective configuration loaded","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","kind":"EndpointPickerConfig"},"configError":"got runtime.Object without object metadata: {FeatureGates: {}, Plugins: [{single-profile-handler/single-profile-handler} {queue-scorer/queue-scorer} {prefix-cache-scorer/prefix-cache-scorer} {max-score-picker/max-score-picker} {fcfs-ordering-policy/fcfs-ordering-policy} {global-strict-fairness-policy/global-strict-fairness-policy}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"runner/runner.go:549","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"setup","caller":"runner/runner.go:301","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"setup","caller":"runner/runner.go:315","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","logger":"setup.SaturationDetector","caller":"utilizationdetector/detector.go:70","msg":"Creating new SaturationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"setup","caller":"runner/runner.go:350","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"setup","caller":"runner/runner.go:644","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"setup","caller":"runner/runner.go:209","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:42Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","reconcileID":"d17cc85d-abde-46db-ad06-94910f9bc52c","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:25:42Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:52Z","caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x","reconcileID":"ddfb8bed-d714-4a8a-a840-4081ea85f7e0"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:52Z","caller":"metrics/pod_metrics.go:76","msg":"Starting refresher","endpoint":{"name":"llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x-rank-0","namespace":"kserve-ci-e2e-test"},"metadata":"{NamespacedName:kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x-rank-0 PodName:llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x Address:10.132.0.47 Port:8000 MetricsHost:10.132.0.47:8000 Labels:map[app.kubernetes.io/component:llminferenceservice-workload app.kubernetes.io/name:llmisvc-router-managed-test-llm-4b931143 app.kubernetes.io/part-of:llminferenceservice kserve.io/component:workload llm-d.ai/role:both pod-template-hash:66f88bc44d]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:25:53Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","reconcileID":"637250b7-3937-462d-bf7e-6f6358c70612","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'tokenizer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 INFO 06-15 06:25:48 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 06-15 06:25:48 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] 2026-06-15 06:25:49,712 [INFO] [root] TokenizationServiceServicer initialized [e2e-llm-inference-service] 2026-06-15 06:25:49,713 [INFO] [root] gRPC reflection disabled (set `ENABLE_GRPC_REFLECTION=1` to enable) [e2e-llm-inference-service] 2026-06-15 06:25:49,713 [INFO] [root] gRPC server configured to listen on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:25:49,713 [INFO] [root] gRPC server started on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:25:49,714 [INFO] [root] Probe server started on port 8082 [e2e-llm-inference-service] 2026-06-15 06:25:49,714 [INFO] [root] Server started. [e2e-llm-inference-service] 2026-06-15 06:25:52,307 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:25:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:25:57,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:25:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:02,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:12,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:12,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:32,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:52,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:26:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:26:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:32 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:52,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:27:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:27:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:02,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:28:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:28:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:02,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:32,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:29:57,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:29:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:27,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:32,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:42,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:52 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:57,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:30:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:12 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:22,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:32 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:57,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:31:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:12,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:32:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:02,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:42,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:52,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:33:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:02 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:12,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:12 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:32,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:42,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:34:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:27,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:42,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:52,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:52 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:35:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:02,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:02 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:12,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:22,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:32,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:52,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:57,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:36:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:22,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:32,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:52 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:37:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:38:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:12,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:52,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:39:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:02,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:12,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:12,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:22,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:27,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:32 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:42,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:52,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:57,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:40:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:02,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:12,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:22,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:42,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:57,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:41:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:52 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:42:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:12,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:22,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:27,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:42,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:42,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:52,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:52 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:57,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:43:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:02,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:44:02 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:12,306 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:44:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:12,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:44:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:22,436 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:44:22 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:27,305 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:44:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:32,435 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:44:32 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 06cf98d2-f6bd-4b3b-98ba-795c99ba79db [e2e-llm-inference-service] resourceVersion: '41337' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.41 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] uid: 8e1b69b4-ea1b-42c3-9cb9-591db3c02643 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: db375547-b6df-4d3f-a774-b39bc22121aa [e2e-llm-inference-service] resourceVersion: '41064' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.47 [e2e-llm-inference-service] nodeName: ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] uid: d0c46224-53bd-4494-b0c7-835a49a21cf7 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d0c46224-53bd-4494-b0c7-835a49a21cf7 [e2e-llm-inference-service] resourceVersion: '41062' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 66f88bc44d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.47/23"],"mac_address":"0a:58:0a:84:00:2f","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.47/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.47\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:2f\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d [e2e-llm-inference-service] uid: 3b204cae-4fb7-4b0e-a6e5-c2274d4de174 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-243 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3b204cae-4fb7-4b0e-a6e5-c2274d4de174"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.47"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-vl9lq [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-vl9lq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] hostIP: 10.0.128.243 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.243 [e2e-llm-inference-service] podIP: 10.132.0.47 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.47 [e2e-llm-inference-service] startTime: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim@sha256:bab162bd25e2ed8b15022387cdb223023aeb33be49476af9f0115c0398fb8ff5 [e2e-llm-inference-service] containerID: cri-o://32a6110a7b59885b076d50ddc1d5fbec6164561b724352dbf2a3ef11973d7444 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-vl9lq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] generateName: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8e1b69b4-ea1b-42c3-9cb9-591db3c02643 [e2e-llm-inference-service] resourceVersion: '41335' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5597d7fd6 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.41/23"],"mac_address":"0a:58:0a:85:00:29","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.41/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.41\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:29\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6 [e2e-llm-inference-service] uid: d87575c6-df17-4fb6-b992-6d245459cce1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-25 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d87575c6-df17-4fb6-b992-6d245459cce1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.41"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-8t9fg [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-8t9fg [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-8t9fg [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-rxpbg [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] hostIP: 10.0.141.25 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.25 [e2e-llm-inference-service] podIP: 10.133.0.41 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.41 [e2e-llm-inference-service] startTime: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-scheduler@sha256:88de279c6eb6758a4c600de9730e49e46b04c392846afedd03d82447379c9e7a [e2e-llm-inference-service] containerID: cri-o://547f599db73de1ca1ad8af0e4f755607186abe4236f58063abf49196fa6337e3 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-8t9fg [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-uds-tokenizer@sha256:aed091a51f3d64458f1fdb451d21f745186bb4517a7ba0c49913a0c617366a3e [e2e-llm-inference-service] containerID: cri-o://0027d74da24955b56dcac9ad15899c3fafd670b7523765b2f48f0c7501b244b3 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-8t9fg [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 75024947-bba0-4553-8488-1bcb7e7aaf11 [e2e-llm-inference-service] resourceVersion: '40798' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-rxpbg [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-rxpbg"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjfwp"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-rxpbg [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-rxpbg [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3ca0efaf-aada-4087-ad06-1ea669727ddc [e2e-llm-inference-service] resourceVersion: '40828' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.47.15 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.47.15 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cd91c299-5a97-41a1-8d41-268f54e26fb5 [e2e-llm-inference-service] resourceVersion: '40791' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.66.224 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.66.224 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 462c0065-8494-4483-86c7-2e6ab2e545d7 [e2e-llm-inference-service] resourceVersion: '41066' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ecee695a-211a-459d-85d4-c556133b52d3 [e2e-llm-inference-service] resourceVersion: '41339' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3b204cae-4fb7-4b0e-a6e5-c2274d4de174 [e2e-llm-inference-service] resourceVersion: '41065' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 66f88bc44d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] uid: 462c0065-8494-4483-86c7-2e6ab2e545d7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"462c0065-8494-4483-86c7-2e6ab2e545d7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 66f88bc44d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 66f88bc44d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d87575c6-df17-4fb6-b992-6d245459cce1 [e2e-llm-inference-service] resourceVersion: '41338' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5597d7fd6 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] uid: ecee695a-211a-459d-85d4-c556133b52d3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ecee695a-211a-459d-85d4-c556133b52d3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5597d7fd6 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5597d7fd6 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5fa442ce-8fd4-44de-ac1e-4777b4a3cdd0 [e2e-llm-inference-service] resourceVersion: '40820' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0881cb7d-e921-4cb1-ada2-64ccb7c40e1f [e2e-llm-inference-service] resourceVersion: '40817' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service-4hgxc [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-4b931143-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6b2d64b6-b04c-45e0-9d49-5120b97687bb [e2e-llm-inference-service] resourceVersion: '41336' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] uid: 3ca0efaf-aada-4087-ad06-1ea669727ddc [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:26:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3ca0efaf-aada-4087-ad06-1ea669727ddc"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.41 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] uid: 8e1b69b4-ea1b-42c3-9cb9-591db3c02643 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-s2fdgv [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 56ba70c5-6bad-4728-a553-aee100f8bfab [e2e-llm-inference-service] resourceVersion: '41063' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] uid: cd91c299-5a97-41a1-8d41-268f54e26fb5 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cd91c299-5a97-41a1-8d41-268f54e26fb5"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.47 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] uid: d0c46224-53bd-4494-b0c7-835a49a21cf7 [e2e-llm-inference-service] nodeName: ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5fa442ce-8fd4-44de-ac1e-4777b4a3cdd0 [e2e-llm-inference-service] resourceVersion: '40820' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0881cb7d-e921-4cb1-ada2-64ccb7c40e1f [e2e-llm-inference-service] resourceVersion: '40817' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:25:54Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '41104' [e2e-llm-inference-service] uid: 951ac285-05b5-4ea4-9b4f-9b8e8d55e2d7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:25:54Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '41104' [e2e-llm-inference-service] uid: 951ac285-05b5-4ea4-9b4f-9b8e8d55e2d7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '41084' [e2e-llm-inference-service] uid: 75a0d492-6388-4fa1-b235-a4a4ba697d39 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:44Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:25:45Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '40987' [e2e-llm-inference-service] uid: a3f1d0f5-a16b-4442-8f07-e5db3c660ea7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:45Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:25:45Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '40872' [e2e-llm-inference-service] uid: 6f38e7f9-3822-40ae-80eb-6b5920aa8e71 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '41090' [e2e-llm-inference-service] uid: 56b551dd-1c59-496c-bf9a-1c01c99ab8d4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '40897' [e2e-llm-inference-service] uid: c48045cb-2dbd-4365-936c-a1159f69f5c2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '40872' [e2e-llm-inference-service] uid: 6f38e7f9-3822-40ae-80eb-6b5920aa8e71 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '41090' [e2e-llm-inference-service] uid: 56b551dd-1c59-496c-bf9a-1c01c99ab8d4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '40897' [e2e-llm-inference-service] uid: c48045cb-2dbd-4365-936c-a1159f69f5c2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:42Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '40872' [e2e-llm-inference-service] uid: 6f38e7f9-3822-40ae-80eb-6b5920aa8e71 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:53Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '41090' [e2e-llm-inference-service] uid: 56b551dd-1c59-496c-bf9a-1c01c99ab8d4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:43Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '40897' [e2e-llm-inference-service] uid: c48045cb-2dbd-4365-936c-a1159f69f5c2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"312964c8-6ff6-402d-98a0-b4443b3dd91f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:25:41Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 312964c8-6ff6-402d-98a0-b4443b3dd91f [e2e-llm-inference-service] resourceVersion: '40840' [e2e-llm-inference-service] uid: 32770c78-950c-435a-88cd-39e49157cf53 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:44:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 66f88bc44d [e2e-llm-inference-service] timestamp: '2026-06-15T06:44:24Z' [e2e-llm-inference-service] window: 15.756s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 9238702n [e2e-llm-inference-service] memory: 25048Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:44:39Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5597d7fd6 [e2e-llm-inference-service] timestamp: '2026-06-15T06:44:21Z' [e2e-llm-inference-service] window: 25.23s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 8131666n [e2e-llm-inference-service] memory: 28324Ki [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 230598n [e2e-llm-inference-service] memory: 361968Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-06-15T06:44:39.550867] end - ❌ 1143.395s: Service returned 503: inference gateway: ServiceUnavailable - failed to find candidate pods for serving the request [e2e-llm-inference-service] _ test_llm_inference_service[router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-with-refs', 'scheduler-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KSer... {'name': 'model-fb-opt-125m-router-with-r-6d64416a'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:723: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...-with-ec5d4bfa'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-6d64416a'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-06-15T06:30:03.487652', start_time = 1781505003.4879394 [e2e-llm-inference-service] duration = 901.1005208492279, timestamp_end = '2026-06-15T06:45:04.588475' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....router-with-ec5d4bfa'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-6d64416a'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f1922fb9ee0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1110: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:34 Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:57 ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1419 ✓ Created/updated Gateway router-gateway-1 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:121 Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:149 Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:157 ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1428 ✓ Created/updated HTTPRoute router-route-1 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:121 Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:149 Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:157 ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1428 ✓ Created/updated HTTPRoute router-route-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-with-refs-router-with-re-997af47d in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-with-refs-router-with-re-997af47d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-with-refs-router-with-re-997af47d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig scheduler-managed-router-with-r-6bb62f6a in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig scheduler-managed-router-with-r-6bb62f6a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig scheduler-managed-router-with-r-6bb62f6a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-single-cpu-router-with-ec5d4bfa in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-router-with-ec5d4bfa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-router-with-ec5d4bfa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig model-fb-opt-125m-router-with-r-6d64416a in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-router-with-r-6d64416a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-router-with-r-6d64416a [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-06-15T06:30:02.188611] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-with-refs', 'scheduler-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='router-with-refs-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway={'apiVersion': 'gateway.networking.k8s.io/v1', 'kind': 'Gateway', 'metadata': {'name': 'router-gateway-1', 'namespace': 'kserve-ci-e2e-test'}, 'spec': {'gatewayClassName': 'openshift-default', 'listeners': [{'name': 'http', 'port': 80, 'protocol': 'HTTP', 'allowedRoutes': {'namespaces': {'from': 'All'}}}]}}, before_test=[ at 0x7f19234a89a0>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'router-with-refs-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-refs-router-with-re-997af47d'}, [e2e-llm-inference-service] {'name': 'scheduler-managed-router-with-r-6bb62f6a'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-router-with-ec5d4bfa'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-6d64416a'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-06-15T06:30:02.266530] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'router-with-refs-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-refs-router-with-re-997af47d'}, [e2e-llm-inference-service] {'name': 'scheduler-managed-router-with-r-6bb62f6a'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-router-with-ec5d4bfa'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-6d64416a'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-06-15T06:30:03.487455] end - ✅ in 1.221s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-06-15T06:30:03.487652] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'router-with-refs-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-refs-router-with-re-997af47d'}, [e2e-llm-inference-service] {'name': 'scheduler-managed-router-with-r-6bb62f6a'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-router-with-ec5d4bfa'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-6d64416a'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:14Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:14Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:14Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:30:14Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:14Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:14Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:14Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:14Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:14Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1130 Timed out waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-06-15T06:45:04.588475] end - ❌ 901.101s: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:742 [router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] ❌ ERROR: Failed to call llm inference service router-with-refs-test: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1151 🔍 # Diagnostics for 'router-with-refs-test' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1152 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1153 # LLMInferenceService router-with-refs-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1156 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:02Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:02Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:03Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:32:23Z' [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '45750' [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-with-refs-router-with-re-997af47d [e2e-llm-inference-service] - name: scheduler-managed-router-with-r-6bb62f6a [e2e-llm-inference-service] - name: workload-single-cpu-router-with-ec5d4bfa [e2e-llm-inference-service] - name: model-fb-opt-125m-router-with-r-6d64416a [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://afa76378251b64e899ef81fa37a9d268-277050783.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/router-with-refs-test [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://afa76378251b64e899ef81fa37a9d268-277050783.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/router-with-refs-test/health [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://router-gateway-1-openshift-default.kserve-ci-e2e-test.svc.cluster.local/kserve-ci-e2e-test/router-with-refs-test [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://router-gateway-1-openshift-default.kserve-ci-e2e-test.svc.cluster.local/kserve-ci-e2e-test/router-with-refs-test/health [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:21Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: GatewaysReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:21Z' [e2e-llm-inference-service] message: 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: [e2e-llm-inference-service] "False" (reason "InvalidKind", message "referencing unsupported backendRef: [e2e-llm-inference-service] group \"inference.networking.x-k8s.io\" kind \"InferencePool\"")]' [e2e-llm-inference-service] reason: HTTPRoutesNotReady [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:21Z' [e2e-llm-inference-service] message: Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool [e2e-llm-inference-service] exists but no Gateway controller has accepted it yet [e2e-llm-inference-service] reason: WaitingForGateway [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:32:23Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:21Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:21Z' [e2e-llm-inference-service] message: 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: [e2e-llm-inference-service] "False" (reason "InvalidKind", message "referencing unsupported backendRef: [e2e-llm-inference-service] group \"inference.networking.x-k8s.io\" kind \"InferencePool\"")]' [e2e-llm-inference-service] reason: HTTPRoutesNotReady [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:21Z' [e2e-llm-inference-service] message: 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: [e2e-llm-inference-service] "False" (reason "InvalidKind", message "referencing unsupported backendRef: [e2e-llm-inference-service] group \"inference.networking.x-k8s.io\" kind \"InferencePool\"")]' [e2e-llm-inference-service] reason: HTTPRoutesNotReady [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:32:23Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://afa76378251b64e899ef81fa37a9d268-277050783.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/router-with-refs-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-699694bb49-m6gc4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-b5799d8f5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-699694bb49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-85d86d876c-vrqhw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" in 3.371s (3.371s including waiting). Image size: 299992506 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6c5d597fbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-85d86d876c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.36s (27.36s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-f5744d7b7-gjb94 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-7748b48dbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-f5744d7b7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:25 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-598d8c75cc-qw9md [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-54bd696fdf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-598d8c75cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-779977f94c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.286s (1.286s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:50 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:07 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:35 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:20 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:00 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:07:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7bc88f48bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-548bd48954 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-67h82 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.023s (1.023s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-67h82 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-h6wcn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.38:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-74dcd66d7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5c556785f6 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.707s (2.707s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:33 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-75dcfd69c9-dh6qf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-75dcfd69c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:59 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:32:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7d4868d689 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-578d595fc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-96f8b89cb-j7r99 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-96f8b89cb-j7r99 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-9c4c7855f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-96f8b89cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.082s (1.082s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 951ms (951ms including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.592s (30.592s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.034s (1.034s including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 31.996s (31.996s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.34:8082/healthz": dial tcp 10.133.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884fbb from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5d7479f884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod router-with-refs-test-kserve-578d595fc-gtvkx (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:30:12.996 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 06:30:12.996 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_802b1ae1-4ac6-4288-89da-1d848e251758'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_7dba593d-854f-4eff-ad7d-537424c0de2a'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_54f1a22b-1332-4929-98ac-ac7b9d957dbd'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_199b8140-2419-461f-a012-b3a9fbb869ea'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/gPcsVCQDYDHk-_n0G9uADl7PXIM=.61c60ec52ed43038fff0fbbd68b080c94b0d94b4c8458dbd65965f9b17631c89.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_27920dff-3921-43d4-af36-68957574279b'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_551272e4-1c7b-4303-9978-dc7e57017745'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_c426ffd0-26dc-4443-a4e6-bef3463d3925'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Q1p2l2BzM1m6P5jKvr8WTq1TUio=.2d74da6615135c58cf3cf9ad4cb11e7c613ff9e55fe658a47ab83b6c8d1174a9.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_1bb1a10f-e116-4d16-b746-2ae1ae69a747'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_75f7263b-fd32-4347-bff3-cd653313e11b'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/a7eHxRFT3OeMBIFg52k2nfj5m7w=.db7090b0c8b34dd957a7e0656c718f978f9203cc874018f37dda44108be5970a.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_b1b43f8a-9e52-4471-ac85-eed10af5edea'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_c3583a8c-0497-4829-86cd-5ede282c7a56'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_96a9771a-9eac-4038-8171-5d9af226ffdb'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-06-15 06:30:23.391 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 06:30:23.391 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 10.394461955999759 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 (APIServer pid=1) DEBUG 06-15 06:34:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:34:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:35:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:36:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:37:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:38:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:39:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:40:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:41:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:42:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:43:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:44:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:45:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:45:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:30:13.462 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 06:30:13.463 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-06-15 06:30:13.463 1 storage.initializer INFO [kserve_storage.py:download():169] Allow patterns: ['tokenizer.json', 'tokenizer_config.json', 'special_tokens_map.json', 'vocab.json', 'merges.txt', 'config.json', 'generation_config.json'] [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e18faa22-fb98-41d5-925d-3da566562d46'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_d4206220-5a9a-45a2-ab49-f0b9b821bab2'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_9a7660d1-e3fd-45a2-a96c-3fba24f3bf17'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_d74827d8-6b4b-4636-bbd5-486950decc6f'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_790f8faa-978f-47fc-b92a-f36628ff3a19'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_1992d66c-79df-4dd6-b9ab-8c6f763b0861'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-06-15 06:30:13.922 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 06:30:13.922 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 0.4595070760001363 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"setup","caller":"runner/runner.go:150","msg":"GIE build","commit-sha":"","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"setup","caller":"runner/runner.go:169","msg":"Flags processed","flags":{"cache-info-metric":"vllm:cache_config_info","cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"kv-cache-usage-percentage-metric":"vllm:kv_cache_usage_perc","lora-info-metric":"vllm:lora_requests_info","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"router-with-refs-test-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"total-queued-requests-metric":"vllm:num_requests_waiting","total-running-requests-metric":"vllm:num_requests_running","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"setup.trace","caller":"tracing/telemetry.go:131","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"loader/configloader.go:65","msg":"Loaded raw configuration","config":"{FeatureGates: {}, Plugins: [{/single-profile-handler} {/queue-scorer} {/prefix-cache-scorer} {/max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"prefix/plugin.go:203","msg":"BlockSize is not positive, using default value","default":16} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"prefix/plugin.go:213","msg":"PrefixCachePlugin initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":256,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"loader/configloader.go:98","msg":"Effective configuration loaded","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","kind":"EndpointPickerConfig"},"configError":"got runtime.Object without object metadata: {FeatureGates: {}, Plugins: [{single-profile-handler/single-profile-handler} {queue-scorer/queue-scorer} {prefix-cache-scorer/prefix-cache-scorer} {max-score-picker/max-score-picker} {fcfs-ordering-policy/fcfs-ordering-policy} {global-strict-fairness-policy/global-strict-fairness-policy}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"runner/runner.go:549","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"setup","caller":"runner/runner.go:301","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"setup","caller":"runner/runner.go:315","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","logger":"setup.SaturationDetector","caller":"utilizationdetector/detector.go:70","msg":"Creating new SaturationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"setup","caller":"runner/runner.go:350","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"setup","caller":"runner/runner.go:644","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"setup","caller":"runner/runner.go:209","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:30:15Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:30:15Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"router-with-refs-test-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"router-with-refs-test-inference-pool","reconcileID":"8058e482-52d5-425e-8ba2-a8ca482dfeae","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:32:22Z","caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"router-with-refs-test-kserve-578d595fc-gtvkx","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"router-with-refs-test-kserve-578d595fc-gtvkx","reconcileID":"5a758992-ad3d-48bc-8e15-1c044b8d2d76"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:32:22Z","caller":"metrics/pod_metrics.go:76","msg":"Starting refresher","endpoint":{"name":"router-with-refs-test-kserve-578d595fc-gtvkx-rank-0","namespace":"kserve-ci-e2e-test"},"metadata":"{NamespacedName:kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx-rank-0 PodName:router-with-refs-test-kserve-578d595fc-gtvkx Address:10.134.0.41 Port:8000 MetricsHost:10.134.0.41:8000 Labels:map[app.kubernetes.io/component:llminferenceservice-workload app.kubernetes.io/name:router-with-refs-test app.kubernetes.io/part-of:llminferenceservice kserve.io/component:workload llm-d.ai/role:both pod-template-hash:578d595fc]}"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'tokenizer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 INFO 06-15 06:30:20 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 06-15 06:30:20 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] 2026-06-15 06:30:22,583 [INFO] [root] TokenizationServiceServicer initialized [e2e-llm-inference-service] 2026-06-15 06:30:22,583 [INFO] [root] gRPC reflection disabled (set `ENABLE_GRPC_REFLECTION=1` to enable) [e2e-llm-inference-service] 2026-06-15 06:30:22,583 [INFO] [root] gRPC server configured to listen on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:30:22,583 [INFO] [root] gRPC server started on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:30:22,584 [INFO] [root] Probe server started on port 8082 [e2e-llm-inference-service] 2026-06-15 06:30:22,584 [INFO] [root] Server started. [e2e-llm-inference-service] 2026-06-15 06:30:22,958 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:30:22 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:23,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:30:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:27,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:30:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:33,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:30:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:30:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:43,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:30:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:30:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:30:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:30:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:03,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:13,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:27,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:33,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:33 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:53 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:31:57,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:31:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:03,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:13,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:23,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:27,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:33,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:43,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:32:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:32:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:03,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:03 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:13,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:23,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:27,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:33,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:33:57,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:33:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:03,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:13,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:23,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:23 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:27,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:33,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:42,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:43,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:34:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:34:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:03,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:13,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:27,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:33,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:43,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:43 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:35:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:35:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:03,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:13,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:13 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:23,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:27,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:33,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:53,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:36:57,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:36:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:03,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:13,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:13 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:27,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:33,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:37:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:37:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:03,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:13,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:27,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:33,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:53,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:38:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:38:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:03,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:13,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:27,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:33,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:33 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:53 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:39:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:39:57 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:03,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:03 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:13,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:27,962 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:27 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:33,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:53,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:40:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:40:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:03,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:13,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:27,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:33,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:42,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:53,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:41:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:41:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:03,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:12,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:13,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:23,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:23 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:27,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:33,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:43,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:53 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:42:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:42:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:03,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:03 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:12,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:12 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:13,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:27,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:33,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:42 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:43 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:53,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:53 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:43:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:43:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:03,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:03 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:12,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:12 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:13,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:13 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:23,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:23 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:27,956 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:27 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:33,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:33 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:42,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:42 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:43,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:53,032 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:53 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:44:57,957 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:44:57 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:45:03,031 [INFO] [aiohttp.access] 10.134.0.2 [15/Jun/2026:06:45:03 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4a4d7e53-1ecf-44d4-9d6d-0f534439dfe7 [e2e-llm-inference-service] resourceVersion: '44796' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.42 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] uid: c837db67-d0c8-447e-87c9-5dec6015f51f [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 26b3643b-76f4-4b17-9d2f-ce137a0c0617 [e2e-llm-inference-service] resourceVersion: '45743' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.41 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] uid: c15f33fe-f482-409c-8011-41250b69694d [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] generateName: router-with-refs-test-kserve-578d595fc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c15f33fe-f482-409c-8011-41250b69694d [e2e-llm-inference-service] resourceVersion: '45742' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 578d595fc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.41/23"],"mac_address":"0a:58:0a:86:00:29","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.41/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.41\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:29\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: router-with-refs-test-kserve-578d595fc [e2e-llm-inference-service] uid: 0d603bab-21f2-4f77-87e7-1041dbaae626 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-226 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0d603bab-21f2-4f77-87e7-1041dbaae626"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.41"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-tdqbj [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-tdqbj [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n ucx_hcas=()\n for hca in \"${active_hcas[@]}\"\ [e2e-llm-inference-service] ; do\n ucx_hcas+=(\"${hca}:1\")\n done\n\n # Check if we found any active\ [e2e-llm-inference-service] \ HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements\ [e2e-llm-inference-service] \ with a comma\n hcas=$(IFS=,; echo \"${active_hcas[*]}\")\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Setting active HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n\ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if counts are equal - prefer lower index number\n\ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8000\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-tdqbj [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:24Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] hostIP: 10.0.128.226 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.226 [e2e-llm-inference-service] podIP: 10.134.0.41 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.41 [e2e-llm-inference-service] startTime: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T06:30:23Z' [e2e-llm-inference-service] containerID: cri-o://dbadf93e026882e53c6bdf1347713790357cd44393bea4f60a643d328ceaf7e4 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://dbadf93e026882e53c6bdf1347713790357cd44393bea4f60a643d328ceaf7e4 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-tdqbj [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:30:24Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0 [e2e-llm-inference-service] containerID: cri-o://8c7c363c0368b1dd791993f3898677a646e12c2af435e2e198adb3ba9c1f22db [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-tdqbj [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] generateName: router-with-refs-test-kserve-router-scheduler-7d4868d689- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c837db67-d0c8-447e-87c9-5dec6015f51f [e2e-llm-inference-service] resourceVersion: '44794' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7d4868d689 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.42/23"],"mac_address":"0a:58:0a:86:00:2a","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.42/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.42\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:2a\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: router-with-refs-test-kserve-router-scheduler-7d4868d689 [e2e-llm-inference-service] uid: dfa69409-5eec-4453-854d-0e1a9b183345 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-226 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfa69409-5eec-4453-854d-0e1a9b183345"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.42"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-4rv7r [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-4rv7r [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - router-with-refs-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-4rv7r [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] - name: kube-api-access-4rv7r [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-test-epp-sa [e2e-llm-inference-service] serviceAccount: router-with-refs-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: router-with-refs-test-epp-sa-dockercfg-5mggr [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:14Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] hostIP: 10.0.128.226 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.226 [e2e-llm-inference-service] podIP: 10.134.0.42 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.42 [e2e-llm-inference-service] startTime: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] containerID: cri-o://baffc1708f3a7efef0c0b049d34c2ca7f8c6aa71d01be02675cd452ffb11b1c6 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://baffc1708f3a7efef0c0b049d34c2ca7f8c6aa71d01be02675cd452ffb11b1c6 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-4rv7r [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:30:15Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-scheduler@sha256:88de279c6eb6758a4c600de9730e49e46b04c392846afedd03d82447379c9e7a [e2e-llm-inference-service] containerID: cri-o://803e4a2f17d5f799d87d17f68a1a177346f5ded01e24303c753de50ba88947ce [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-4rv7r [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:30:15Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-uds-tokenizer@sha256:aed091a51f3d64458f1fdb451d21f745186bb4517a7ba0c49913a0c617366a3e [e2e-llm-inference-service] containerID: cri-o://f44c996a236d3e1910dee3fb14563f4bc2dc932284021163b6def597a43b3614 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-4rv7r [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0c9f14ff-0819-41d6-9091-a3e4e29df584 [e2e-llm-inference-service] resourceVersion: '44277' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: router-with-refs-test-epp-sa-dockercfg-5mggr [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"router-with-refs-test-epp-sa-dockercfg-5mggr"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjfwp"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: router-with-refs-test-epp-sa-dockercfg-5mggr [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: router-with-refs-test-epp-sa-dockercfg-5mggr [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cf2fdb01-f807-46f1-9a5d-9344a86b7de9 [e2e-llm-inference-service] resourceVersion: '44319' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.136.167 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.136.167 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: aff188c0-7112-482c-bce4-75082d4ea50e [e2e-llm-inference-service] resourceVersion: '44265' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.197.118 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.197.118 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e8712d19-620c-43db-9ffb-79505eadcfbc [e2e-llm-inference-service] resourceVersion: '45746' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "router-with-refs-test-kserve-578d595fc" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d5c4411e-483f-461f-922d-99d7ba299302 [e2e-llm-inference-service] resourceVersion: '44798' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - router-with-refs-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-test-epp-sa [e2e-llm-inference-service] serviceAccount: router-with-refs-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "router-with-refs-test-kserve-router-scheduler-7d4868d689" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-578d595fc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0d603bab-21f2-4f77-87e7-1041dbaae626 [e2e-llm-inference-service] resourceVersion: '45745' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 578d595fc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: router-with-refs-test-kserve [e2e-llm-inference-service] uid: e8712d19-620c-43db-9ffb-79505eadcfbc [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e8712d19-620c-43db-9ffb-79505eadcfbc"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 578d595fc [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 578d595fc [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-router-scheduler-7d4868d689 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: dfa69409-5eec-4453-854d-0e1a9b183345 [e2e-llm-inference-service] resourceVersion: '44797' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7d4868d689 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] uid: d5c4411e-483f-461f-922d-99d7ba299302 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5c4411e-483f-461f-922d-99d7ba299302"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7d4868d689 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7d4868d689 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - router-with-refs-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-test-epp-sa [e2e-llm-inference-service] serviceAccount: router-with-refs-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e802d034-bcd9-492b-b276-9315bda35b0f [e2e-llm-inference-service] resourceVersion: '44293' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: router-with-refs-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: router-with-refs-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 36c7889e-b2af-4c6d-bb42-4539508ebfe0 [e2e-llm-inference-service] resourceVersion: '44290' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-epp-service-jw98j [e2e-llm-inference-service] generateName: router-with-refs-test-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cb8971c6-100f-4a65-8c9e-017c82e504cb [e2e-llm-inference-service] resourceVersion: '44795' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: router-with-refs-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: router-with-refs-test-epp-service [e2e-llm-inference-service] uid: cf2fdb01-f807-46f1-9a5d-9344a86b7de9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cf2fdb01-f807-46f1-9a5d-9344a86b7de9"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.42 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] uid: c837db67-d0c8-447e-87c9-5dec6015f51f [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-workload-svc-fjh75 [e2e-llm-inference-service] generateName: router-with-refs-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 663ff8e6-df87-4eae-a3fb-fd8472be2cc7 [e2e-llm-inference-service] resourceVersion: '45744' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] uid: aff188c0-7112-482c-bce4-75082d4ea50e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:32:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"aff188c0-7112-482c-bce4-75082d4ea50e"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.41 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] uid: c15f33fe-f482-409c-8011-41250b69694d [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e802d034-bcd9-492b-b276-9315bda35b0f [e2e-llm-inference-service] resourceVersion: '44293' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:router-with-refs-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 36c7889e-b2af-4c6d-bb42-4539508ebfe0 [e2e-llm-inference-service] resourceVersion: '44290' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] name: router-with-refs-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44324' [e2e-llm-inference-service] uid: 401040d1-69ab-47a2-833a-8178c527cc95 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: router-with-refs-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: {} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:03Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:03Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:30:05Z' [e2e-llm-inference-service] name: router-route-1-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44183' [e2e-llm-inference-service] uid: 1d3354a6-434e-418c-8463-1ee3e5c46bbf [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: router-route-1 [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:03Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:05Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:03Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:03Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:30:05Z' [e2e-llm-inference-service] name: router-route-2-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44180' [e2e-llm-inference-service] uid: f4a4c333-5ddc-4430-b5a9-7de76bfc518e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: router-route-2 [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:04Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:30:05Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] name: router-with-refs-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44331' [e2e-llm-inference-service] uid: 5d0a9db2-d08d-44de-83e8-6ad8512a72a3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] name: router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44340' [e2e-llm-inference-service] uid: edad5e56-35cf-4442-89fc-43e2b88778e5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] name: router-with-refs-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44331' [e2e-llm-inference-service] uid: 5d0a9db2-d08d-44de-83e8-6ad8512a72a3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] name: router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44340' [e2e-llm-inference-service] uid: edad5e56-35cf-4442-89fc-43e2b88778e5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] name: router-with-refs-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44331' [e2e-llm-inference-service] uid: 5d0a9db2-d08d-44de-83e8-6ad8512a72a3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:13Z' [e2e-llm-inference-service] name: router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44340' [e2e-llm-inference-service] uid: edad5e56-35cf-4442-89fc-43e2b88778e5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad907ea1-a6c8-4368-bf84-c54b42fe3fe3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:30:12Z' [e2e-llm-inference-service] name: router-with-refs-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-test [e2e-llm-inference-service] uid: ad907ea1-a6c8-4368-bf84-c54b42fe3fe3 [e2e-llm-inference-service] resourceVersion: '44328' [e2e-llm-inference-service] uid: 54a726a3-72b8-42c5-899d-36656e6f4971 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: router-with-refs-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:05Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 578d595fc [e2e-llm-inference-service] timestamp: '2026-06-15T06:44:43Z' [e2e-llm-inference-service] window: 16.553s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 100238869n [e2e-llm-inference-service] memory: 2404136Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:05Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7d4868d689 [e2e-llm-inference-service] timestamp: '2026-06-15T06:44:36Z' [e2e-llm-inference-service] window: 18.836s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 58201157n [e2e-llm-inference-service] memory: 28900Ki [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 264984n [e2e-llm-inference-service] memory: 359956Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-06-15T06:45:06.106435] end - ❌ 903.917s: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:30:21Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-1: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:30:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:32:23Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt='KServe is a', ...opt-125m-with-lora-hf-a7886ead'}]}, [e2e-llm-inference-service] 'status': None}, model_name='publishers/kserve-ci-e2e-test/models/lora-adapter-1') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:723: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...vc-mod-495991f8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-a7886ead'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-06-15T06:44:39.921048', start_time = 1781505879.9213114 [e2e-llm-inference-service] duration = 900.540575504303, timestamp_end = '2026-06-15T06:59:40.461894' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....-llmisvc-mod-495991f8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-a7886ead'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f7425e99d00> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1110: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-fb-98f275aa in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-fb-98f275aa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-fb-98f275aa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-495991f8 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-495991f8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-495991f8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-a7886ead in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-a7886ead [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-a7886ead [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-06-15T06:44:39.798748] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt='KServe is a', service_name='llmisvc-model-fb-opt-125m-with-7ca60146', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f7426d2ff60>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/lora-adapter-1'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-7ca60146', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-98f275aa'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-495991f8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-a7886ead'}]}, [e2e-llm-inference-service] 'status': None}, model_name='publishers/kserve-ci-e2e-test/models/lora-adapter-1')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-06-15T06:44:39.811100] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-7ca60146', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-98f275aa'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-495991f8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-a7886ead'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-06-15T06:44:39.920966] end - ✅ in 0.110s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-06-15T06:44:39.921048] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-7ca60146', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-98f275aa'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-495991f8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-a7886ead'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:45:16Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1130 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-06-15T06:59:40.461894] end - ❌ 900.541s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:742 [router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf] ❌ ERROR: Failed to call llm inference service llmisvc-model-fb-opt-125m-with-7ca60146: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1151 🔍 # Diagnostics for 'llmisvc-model-fb-opt-125m-with-7ca60146' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1152 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1153 # LLMInferenceService llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1156 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:44:39Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:44:39Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:44:39Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:45:52Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '54811' [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-fb-98f275aa [e2e-llm-inference-service] - name: workload-single-cpu-llmisvc-mod-495991f8 [e2e-llm-inference-service] - name: model-fb-opt-125m-with-lora-hf-a7886ead [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:40Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:40Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:40Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:16Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:40Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:52Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:52Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:40Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-699694bb49-m6gc4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-b5799d8f5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-699694bb49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-85d86d876c-vrqhw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" in 3.371s (3.371s including waiting). Image size: 299992506 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6c5d597fbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-85d86d876c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.36s (27.36s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-f5744d7b7-gjb94 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-7748b48dbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-f5744d7b7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.46:8001/health": dial tcp 10.134.0.46:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6b5b6588r7 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-6b5b6588r7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-6b5b695dd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-78b45dc7ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:53 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:25 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-598d8c75cc-qw9md [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-54bd696fdf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-598d8c75cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-779977f94c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.286s (1.286s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 867ms (867ms including waiting). Image size: 67767940 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.44:8001/health": dial tcp 10.134.0.44:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8649d9d4d8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:50 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:07 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:35 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:20 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-7cdd64995b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:16 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:16 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:00 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:07:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6jhfg to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6jhfg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-57bd5888f4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7bc88f48bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-548bd48954 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-67h82 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.023s (1.023s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-67h82 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-h6wcn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.38:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-74dcd66d7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5c556785f6 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.707s (2.707s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:33 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-75dcfd69c9-dh6qf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-75dcfd69c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:59 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-78c98f6f4c-ddrqp to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:12 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:14 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.491s (2.491s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-78c98f6f4c-ddrqp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-78c98f6f4c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:15 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:31 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:54:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.48:8001/health": dial tcp 10.134.0.48:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-6f78896447-wshh4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:55:06 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-5fc8578dd5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-6f78896447 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:32:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7d4868d689 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-578d595fc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-96f8b89cb-j7r99 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-96f8b89cb-j7r99 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-9c4c7855f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-96f8b89cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.082s (1.082s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 951ms (951ms including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.592s (30.592s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.034s (1.034s including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 31.996s (31.996s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.34:8082/healthz": dial tcp 10.133.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884fbb from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5d7479f884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:45:14.410 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models'), ('hf://edbeeching/opt-125m-lora', '/mnt/lora/lora-adapter-1')] [e2e-llm-inference-service] 2026-06-15 06:45:14.411 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '9877d360-6036-4d35-8e92-b3c4888a7cbf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 15 Jun 2026 06:59:41 GMT', 'Content-Length': '245'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:45:14.650 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 06:45:14.650 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-06-15 06:45:14.650 1 storage.initializer INFO [kserve_storage.py:download():169] Allow patterns: ['tokenizer.json', 'tokenizer_config.json', 'special_tokens_map.json', 'vocab.json', 'merges.txt', 'config.json', 'generation_config.json'] [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_b1930608-e8e5-48c7-959d-b0aa1fe7c954'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_9a83c601-a8cf-4357-9939-e569feb66089'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_fdfebf00-94e1-4d43-825d-24183ed7fb8e'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_77e37e44-d4c0-4023-a6d2-069bbe7ff267'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_c8c04ecd-ef93-4486-ba03-f7fa6e444936'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_96aaa6c0-f662-460f-bf24-17e6c70065ad'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-06-15 06:45:15.109 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 06:45:15.109 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 0.4592664690003403 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":"2026-06-15T06:45:15Z","logger":"setup","caller":"runner/runner.go:150","msg":"GIE build","commit-sha":"","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:15Z","logger":"setup","caller":"runner/runner.go:169","msg":"Flags processed","flags":{"cache-info-metric":"vllm:cache_config_info","cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"kv-cache-usage-percentage-metric":"vllm:kv_cache_usage_perc","lora-info-metric":"vllm:lora_requests_info","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"total-queued-requests-metric":"vllm:num_requests_waiting","total-running-requests-metric":"vllm:num_requests_running","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:15Z","logger":"setup.trace","caller":"tracing/telemetry.go:131","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:15Z","caller":"loader/configloader.go:65","msg":"Loaded raw configuration","config":"{FeatureGates: {}, Plugins: [{/single-profile-handler} {/queue-scorer} {/prefix-cache-scorer} {/max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"prefix/plugin.go:203","msg":"BlockSize is not positive, using default value","default":16} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"prefix/plugin.go:213","msg":"PrefixCachePlugin initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":256,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"loader/configloader.go:98","msg":"Effective configuration loaded","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","kind":"EndpointPickerConfig"},"configError":"got runtime.Object without object metadata: {FeatureGates: {}, Plugins: [{single-profile-handler/single-profile-handler} {queue-scorer/queue-scorer} {prefix-cache-scorer/prefix-cache-scorer} {max-score-picker/max-score-picker} {fcfs-ordering-policy/fcfs-ordering-policy} {global-strict-fairness-policy/global-strict-fairness-policy}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"runner/runner.go:549","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","logger":"setup","caller":"runner/runner.go:301","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","logger":"setup","caller":"runner/runner.go:315","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","logger":"setup.SaturationDetector","caller":"utilizationdetector/detector.go:70","msg":"Creating new SaturationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","logger":"setup","caller":"runner/runner.go:350","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","logger":"setup","caller":"runner/runner.go:644","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","logger":"setup","caller":"runner/runner.go:209","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:16Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool","reconcileID":"6fa81216-666d-4529-b5ef-2057989d9011","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:45:16Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:45:38Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool","reconcileID":"784bb8aa-2e09-43ad-a8bb-6c17753189dd","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'tokenizer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 INFO 06-15 06:45:21 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 06-15 06:45:21 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] 2026-06-15 06:45:23,480 [INFO] [root] TokenizationServiceServicer initialized [e2e-llm-inference-service] 2026-06-15 06:45:23,480 [INFO] [root] gRPC reflection disabled (set `ENABLE_GRPC_REFLECTION=1` to enable) [e2e-llm-inference-service] 2026-06-15 06:45:23,480 [INFO] [root] gRPC server configured to listen on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:45:23,481 [INFO] [root] gRPC server started on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:45:23,481 [INFO] [root] Probe server started on port 8082 [e2e-llm-inference-service] 2026-06-15 06:45:23,481 [INFO] [root] Server started. [e2e-llm-inference-service] 2026-06-15 06:45:24,145 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:45:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:45:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:45:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:45:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:45:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:45:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:45:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:45:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:45:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:45:44,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:45:44 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:45:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:45:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:45:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:45:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:04,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:29 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:34 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:44 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:44,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:54,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:46:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:46:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:14 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:14 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:34 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:54,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:47:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:47:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:04,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:14,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:29 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:54,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:48:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:48:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:04 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:44 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:49:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:49:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:04,955 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:50:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:50:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:24,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:44 +0000] "GET /healthz HTTP/1.1" 200 261 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:51:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:51:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:14 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:44,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:14 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:29,150 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:44,143 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:44 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:04 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:44 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:59,143 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:04 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:14 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:24,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:29,143 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:44,143 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:54,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:04 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:14 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:54 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:04 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:14,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:24,953 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:24 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:34 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:44,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:44 +0000] "GET /healthz HTTP/1.1" 200 261 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:44,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:44 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:54,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:54 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:59,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:04,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:04 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:14,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:14,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:14 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:24,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:24 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:29,144 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:34,954 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:34 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a6909865-9ac2-48ee-8a70-b1ab4ee9cba8 [e2e-llm-inference-service] resourceVersion: '54712' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.42 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw [e2e-llm-inference-service] uid: d15a90cc-1bc7-4614-9bf0-26231da0604c [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9d2ba7fb-4016-45b4-9c32-373097c9d3d8 [e2e-llm-inference-service] resourceVersion: '53992' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - notReadyAddresses: [e2e-llm-inference-service] - ip: 10.134.0.43 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w [e2e-llm-inference-service] uid: 2bcc7d6e-7945-4c51-910d-4a0ffe93df64 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2bcc7d6e-7945-4c51-910d-4a0ffe93df64 [e2e-llm-inference-service] resourceVersion: '53991' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 6dbc7ddb8d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.43/23"],"mac_address":"0a:58:0a:86:00:2b","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.43/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.43\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:2b\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d [e2e-llm-inference-service] uid: 2a8b4d6b-21fa-4755-bad0-3501d8b92cfb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-226 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2a8b4d6b-21fa-4755-bad0-3501d8b92cfb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.43"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-pq8qp [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] - name: kube-api-access-pq8qp [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n ucx_hcas=()\n for hca in \"${active_hcas[@]}\"\ [e2e-llm-inference-service] ; do\n ucx_hcas+=(\"${hca}:1\")\n done\n\n # Check if we found any active\ [e2e-llm-inference-service] \ HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements\ [e2e-llm-inference-service] \ with a comma\n hcas=$(IFS=,; echo \"${active_hcas[*]}\")\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Setting active HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n\ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if counts are equal - prefer lower index number\n\ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8000\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: kube-api-access-pq8qp [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [storage-initializer]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] hostIP: 10.0.128.226 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.226 [e2e-llm-inference-service] podIP: 10.134.0.43 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.43 [e2e-llm-inference-service] startTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://47a49ea18aded36d93af1bed5e257738fc523bad3897e0d9cb2050d828fda45d [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-pq8qp [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-pq8qp [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-7cdd64995b- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d15a90cc-1bc7-4614-9bf0-26231da0604c [e2e-llm-inference-service] resourceVersion: '54710' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7cdd64995b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.42/23"],"mac_address":"0a:58:0a:85:00:2a","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.42/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.42\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:2a\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-7cdd64995b [e2e-llm-inference-service] uid: fcaa5ab9-e3b2-4235-bd26-1a7bd182a576 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-25 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"fcaa5ab9-e3b2-4235-bd26-1a7bd182a576"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.42"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-xwz6g [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-xwz6g [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-xwz6g [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] - name: kube-api-access-xwz6g [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-sd6cp [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:15Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] hostIP: 10.0.141.25 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.25 [e2e-llm-inference-service] podIP: 10.133.0.42 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.42 [e2e-llm-inference-service] startTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T06:45:15Z' [e2e-llm-inference-service] containerID: cri-o://81f8edcb9db283dbcea930e9d7dbde3e12ab040f916194724bdf612f9a6f3119 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://81f8edcb9db283dbcea930e9d7dbde3e12ab040f916194724bdf612f9a6f3119 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-xwz6g [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:45:15Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-scheduler@sha256:88de279c6eb6758a4c600de9730e49e46b04c392846afedd03d82447379c9e7a [e2e-llm-inference-service] containerID: cri-o://7d276bd5f209a7e7c71ba25011b21543dcc017333731b89982cb26b3bdfdd24e [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-xwz6g [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:45:16Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-uds-tokenizer@sha256:aed091a51f3d64458f1fdb451d21f745186bb4517a7ba0c49913a0c617366a3e [e2e-llm-inference-service] containerID: cri-o://34fb32fa06720a4f94fef5476a3ceabdbbd1523fa4b06331251b4b32b4151219 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-xwz6g [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ab18f19a-25df-45f6-b85a-b046e34eccca [e2e-llm-inference-service] resourceVersion: '53916' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-sd6cp [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-sd6cp"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjfwp"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-sd6cp [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-sd6cp [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5b7bc502-b79c-4609-9028-06a7872db95e [e2e-llm-inference-service] resourceVersion: '53945' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.86.180 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.86.180 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f17ee831-b321-4ac2-8e1e-fb77096a13a2 [e2e-llm-inference-service] resourceVersion: '53907' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.230.242 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.230.242 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1bfd7251-d9a8-44ba-bcbe-387027ab2792 [e2e-llm-inference-service] resourceVersion: '63001' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:55:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:55:14Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:55:14Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0360faab-ce30-44dd-83be-0cb53b27c9f9 [e2e-llm-inference-service] resourceVersion: '54714' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-7cdd64995b" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2a8b4d6b-21fa-4755-bad0-3501d8b92cfb [e2e-llm-inference-service] resourceVersion: '53927' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 6dbc7ddb8d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] uid: 1bfd7251-d9a8-44ba-bcbe-387027ab2792 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1bfd7251-d9a8-44ba-bcbe-387027ab2792"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 6dbc7ddb8d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 6dbc7ddb8d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-7cdd64995b [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: fcaa5ab9-e3b2-4235-bd26-1a7bd182a576 [e2e-llm-inference-service] resourceVersion: '54713' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7cdd64995b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] uid: 0360faab-ce30-44dd-83be-0cb53b27c9f9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0360faab-ce30-44dd-83be-0cb53b27c9f9"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7cdd64995b [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7cdd64995b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4757cf12-f283-4270-b6ad-ce93d350869b [e2e-llm-inference-service] resourceVersion: '53937' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1fa2cffc-2215-4bba-bcd3-c7a949753d6a [e2e-llm-inference-service] resourceVersion: '53933' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service-phc9x [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0ba25216-977f-42a3-84f0-e425b9798c25 [e2e-llm-inference-service] resourceVersion: '54711' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] uid: 5b7bc502-b79c-4609-9028-06a7872db95e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5b7bc502-b79c-4609-9028-06a7872db95e"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.42 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw [e2e-llm-inference-service] uid: d15a90cc-1bc7-4614-9bf0-26231da0604c [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-sv5pf7f [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d6326a5b-4e70-4135-9398-6e75618fe874 [e2e-llm-inference-service] resourceVersion: '53993' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] uid: f17ee831-b321-4ac2-8e1e-fb77096a13a2 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f17ee831-b321-4ac2-8e1e-fb77096a13a2"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.43 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] serving: false [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w [e2e-llm-inference-service] uid: 2bcc7d6e-7945-4c51-910d-4a0ffe93df64 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4757cf12-f283-4270-b6ad-ce93d350869b [e2e-llm-inference-service] resourceVersion: '53937' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1fa2cffc-2215-4bba-bcd3-c7a949753d6a [e2e-llm-inference-service] resourceVersion: '53933' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:45:39Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '54621' [e2e-llm-inference-service] uid: 2a790a13-9551-4bc9-b08d-a2c158399d8c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:45:39Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '54621' [e2e-llm-inference-service] uid: 2a790a13-9551-4bc9-b08d-a2c158399d8c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '54595' [e2e-llm-inference-service] uid: 5c62ef3b-9cd0-4646-8243-9ae6603c0f51 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:17Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:17Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:45:19Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '54141' [e2e-llm-inference-service] uid: 85fe2bf4-eb78-47b8-9fcf-9968babc8d2f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:18Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:45:19Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '53975' [e2e-llm-inference-service] uid: 3fd5a184-dd30-4d2c-ad7c-d102afe0b30f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '54607' [e2e-llm-inference-service] uid: 787cd36b-3c98-4283-9345-07edba25f7df [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-inference-p-ip-16c62f55.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '53989' [e2e-llm-inference-service] uid: 88906234-0d17-46c1-8d72-54523223f3f6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '53975' [e2e-llm-inference-service] uid: 3fd5a184-dd30-4d2c-ad7c-d102afe0b30f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '54607' [e2e-llm-inference-service] uid: 787cd36b-3c98-4283-9345-07edba25f7df [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-inference-p-ip-16c62f55.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '53989' [e2e-llm-inference-service] uid: 88906234-0d17-46c1-8d72-54523223f3f6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '53975' [e2e-llm-inference-service] uid: 3fd5a184-dd30-4d2c-ad7c-d102afe0b30f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:38Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '54607' [e2e-llm-inference-service] uid: 787cd36b-3c98-4283-9345-07edba25f7df [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-inference-p-ip-16c62f55.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:14Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '53989' [e2e-llm-inference-service] uid: 88906234-0d17-46c1-8d72-54523223f3f6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7af182f6-30ac-4572-9fb1-9e4811eec6cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:45:13Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] uid: 7af182f6-30ac-4572-9fb1-9e4811eec6cd [e2e-llm-inference-service] resourceVersion: '53958' [e2e-llm-inference-service] uid: be12e127-7d0e-4a0f-947e-0c847cdd3d70 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:42Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-7ca60146 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7cdd64995b [e2e-llm-inference-service] timestamp: '2026-06-15T06:59:27Z' [e2e-llm-inference-service] window: 13.069s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 676376n [e2e-llm-inference-service] memory: 22680Ki [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 216542n [e2e-llm-inference-service] memory: 362056Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-06-15T06:59:42.352148] end - ❌ 902.553s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:45:52Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:45:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _ test_llm_inference_service[router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-with-refs-pd', 'scheduler-managed', 'workload-pd-cpu', 'model-fb-opt-125m'], prompt='You a... {'name': 'model-fb-opt-125m-router-with-r-c22ea8a0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:723: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...h-ref-d1f07093'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-c22ea8a0'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-06-15T06:52:14.786869', start_time = 1781506334.7871306 [e2e-llm-inference-service] duration = 900.4981956481934, timestamp_end = '2026-06-15T07:07:15.285340' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....er-with-ref-d1f07093'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-c22ea8a0'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f1922f6b240> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:54:37Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:53:00Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1110: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:34 Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:62 Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:70 ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1419 ✓ Created/updated Gateway router-gateway-2 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:121 Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:149 Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:157 ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1428 ✓ Created/updated HTTPRoute router-route-3 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:121 Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:149 Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] INFO kserve.trace:gw_api.py:157 ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1428 ✓ Created/updated HTTPRoute router-route-4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-with-refs-pd-router-with-c2ec731e in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-with-refs-pd-router-with-c2ec731e [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-with-refs-pd-router-with-c2ec731e [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig scheduler-managed-router-with-r-57d1c131 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig scheduler-managed-router-with-r-57d1c131 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig scheduler-managed-router-with-r-57d1c131 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-pd-cpu-router-with-ref-d1f07093 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-router-with-ref-d1f07093 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-router-with-ref-d1f07093 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig model-fb-opt-125m-router-with-r-c22ea8a0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-router-with-r-c22ea8a0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-router-with-r-c22ea8a0 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-06-15T06:52:14.187536] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-with-refs-pd', 'scheduler-managed', 'workload-pd-cpu', 'model-fb-opt-125m'], prompt='You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.', service_name='router-with-refs-pd-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway={'apiVersion': 'gateway.networking.k8s.io/v1', 'kind': 'Gateway', 'metadata': {'name': 'router-gateway-2', 'namespace': 'kserve-ci-e2e-test'}, 'spec': {'gatewayClassName': 'openshift-default', 'listeners': [{'name': 'http', 'port': 80, 'protocol': 'HTTP', 'allowedRoutes': {'namespaces': {'from': 'All'}}}]}}, before_test=[ at 0x7f19234a8a40>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'router-with-refs-pd-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-refs-pd-router-with-c2ec731e'}, [e2e-llm-inference-service] {'name': 'scheduler-managed-router-with-r-57d1c131'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-router-with-ref-d1f07093'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-c22ea8a0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-06-15T06:52:14.200228] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'router-with-refs-pd-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-refs-pd-router-with-c2ec731e'}, [e2e-llm-inference-service] {'name': 'scheduler-managed-router-with-r-57d1c131'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-router-with-ref-d1f07093'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-c22ea8a0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-06-15T06:52:14.786614] end - ✅ in 0.586s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-06-15T06:52:14.786869] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'router-with-refs-pd-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-with-refs-pd-router-with-c2ec731e'}, [e2e-llm-inference-service] {'name': 'scheduler-managed-router-with-r-57d1c131'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-router-with-ref-d1f07093'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-router-with-r-c22ea8a0'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:52:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:52:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:53:00Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:54:37Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:53:00Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:54:37Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:53:00Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1130 Timed out waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:54:37Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:53:00Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-06-15T07:07:15.285340] end - ❌ 900.498s: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:54:37Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:53:00Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:742 [router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] ❌ ERROR: Failed to call llm inference service router-with-refs-pd-test: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:54:37Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:53:00Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1151 🔍 # Diagnostics for 'router-with-refs-pd-test' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1152 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1153 # LLMInferenceService router-with-refs-pd-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1156 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:14Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:14Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:15Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:55:17Z' [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '63050' [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-with-refs-pd-router-with-c2ec731e [e2e-llm-inference-service] - name: scheduler-managed-router-with-r-57d1c131 [e2e-llm-inference-service] - name: workload-pd-cpu-router-with-ref-d1f07093 [e2e-llm-inference-service] - name: model-fb-opt-125m-router-with-r-c22ea8a0 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ada0b0328a9f745d7906832782092fbe-1076799054.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/router-with-refs-pd-test [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ada0b0328a9f745d7906832782092fbe-1076799054.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/router-with-refs-pd-test/health [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://router-gateway-2-openshift-default.kserve-ci-e2e-test.svc.cluster.local/kserve-ci-e2e-test/router-with-refs-pd-test [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://router-gateway-2-openshift-default.kserve-ci-e2e-test.svc.cluster.local/kserve-ci-e2e-test/router-with-refs-pd-test/health [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:32Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: GatewaysReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:32Z' [e2e-llm-inference-service] message: 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: [e2e-llm-inference-service] "False" (reason "InvalidKind", message "referencing unsupported backendRef: [e2e-llm-inference-service] group \"inference.networking.x-k8s.io\" kind \"InferencePool\"")]' [e2e-llm-inference-service] reason: HTTPRoutesNotReady [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:32Z' [e2e-llm-inference-service] message: Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] exists but no Gateway controller has accepted it yet [e2e-llm-inference-service] reason: WaitingForGateway [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:54:37Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:55:17Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:32Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:32Z' [e2e-llm-inference-service] message: 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: [e2e-llm-inference-service] "False" (reason "InvalidKind", message "referencing unsupported backendRef: [e2e-llm-inference-service] group \"inference.networking.x-k8s.io\" kind \"InferencePool\"")]' [e2e-llm-inference-service] reason: HTTPRoutesNotReady [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:32Z' [e2e-llm-inference-service] message: 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: [e2e-llm-inference-service] "False" (reason "InvalidKind", message "referencing unsupported backendRef: [e2e-llm-inference-service] group \"inference.networking.x-k8s.io\" kind \"InferencePool\"")]' [e2e-llm-inference-service] reason: HTTPRoutesNotReady [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:55:17Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ada0b0328a9f745d7906832782092fbe-1076799054.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/router-with-refs-pd-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-699694bb49-m6gc4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-b5799d8f5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-699694bb49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-85d86d876c-vrqhw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" in 3.371s (3.371s including waiting). Image size: 299992506 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6c5d597fbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-85d86d876c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.36s (27.36s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-f5744d7b7-gjb94 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-7748b48dbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-f5744d7b7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.46:8001/health": dial tcp 10.134.0.46:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6b5b6588r7 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-6b5b6588r7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-6b5b695dd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-78b45dc7ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:53 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:25 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-598d8c75cc-qw9md [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-54bd696fdf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-598d8c75cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-779977f94c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.286s (1.286s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 867ms (867ms including waiting). Image size: 67767940 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.44:8001/health": dial tcp 10.134.0.44:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8649d9d4d8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:50 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:07 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:35 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:20 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-7cdd64995b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:16 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:16 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:00 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:01:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.50:8000/health": dial tcp 10.134.0.50:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-86f69d9999 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:07:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6jhfg to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6jhfg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-57bd5888f4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7bc88f48bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-548bd48954 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-67h82 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.023s (1.023s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-67h82 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-h6wcn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.38:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-74dcd66d7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5c556785f6 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.707s (2.707s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:33 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-75dcfd69c9-dh6qf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-75dcfd69c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:59 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-78c98f6f4c-ddrqp to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:12 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:14 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.491s (2.491s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-78c98f6f4c-ddrqp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-78c98f6f4c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:15 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:31 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:54:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.48:8001/health": dial tcp 10.134.0.48:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-6f78896447-wshh4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:55:06 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-5fc8578dd5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-6f78896447 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:32:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7d4868d689 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-578d595fc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-96f8b89cb-j7r99 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-96f8b89cb-j7r99 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-9c4c7855f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-96f8b89cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.082s (1.082s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 951ms (951ms including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.592s (30.592s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.034s (1.034s including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 31.996s (31.996s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.34:8082/healthz": dial tcp 10.133.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884fbb from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5d7479f884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod router-with-refs-pd-test-kserve-6f78896447-wshh4 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 Flag --decoder-use-tls has been deprecated, use --enable-tls=decoder instead [e2e-llm-inference-service] Flag --prefiller-use-tls has been deprecated, use --enable-tls=prefiller instead [e2e-llm-inference-service] {"level":"info","ts":1781506346.8123612,"msg":"Initializing OpenTelemetry tracing","endpoint":"localhost:4317","service":"llm-d-inference-scheduler"} [e2e-llm-inference-service] {"level":"info","ts":1781506346.812798,"msg":"Configuring trace sampling","ratio":0.1} [e2e-llm-inference-service] {"level":"info","ts":1781506346.8128989,"msg":"OpenTelemetry tracing initialized successfully"} [e2e-llm-inference-service] {"level":"info","ts":1781506346.812909,"msg":"Proxy starting","Built on":"","From Git SHA":"unknown"} [e2e-llm-inference-service] {"level":"info","ts":1781506346.812913,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"EnablePrefillerSampling\":false,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"kserve-ci-e2e-test\",\"InferencePoolName\":\"router-with-refs-pd-test-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1781506346.813382,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"kserve-ci-e2e-test","poolName":"router-with-refs-pd-test-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1781506346.9142907,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1781506346.9154282,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1781506346.9154427,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1781506348.8935266,"logger":"allowlist-validator","msg":"InferencePool added","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506348.906987,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506348.907065,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506351.980523,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506352.9896715,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506356.6015813,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506376.830619,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506376.844384,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506376.8444479,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506398.2555885,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506399.2655084,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506406.8316622,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506406.8459413,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506406.846363,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506436.8323512,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] 2026/06/15 06:53:56 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506436.8456752,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506436.845874,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506466.8323781,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] 2026/06/15 06:54:26 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506466.8495727,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506466.8496308,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506476.5928092,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506476.6120965,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:54:51 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506496.832837,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506496.8456454,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506496.8457038,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:55:06 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506516.6900537,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506516.7070122,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:55:21 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506526.8338354,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506526.8471785,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506526.8472412,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:55:36 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:55:51 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506556.8340774,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506556.8462448,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506556.846306,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:56:01 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:56:11 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:56:21 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506586.8346164,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506586.846819,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506586.8468778,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:56:31 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:56:46 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506616.8349807,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506616.8467693,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506616.8468308,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:56:56 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:57:06 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:57:21 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506646.8350415,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506646.8471575,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506646.8472152,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:57:31 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:57:46 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506676.8357763,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506676.8477297,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506676.8477886,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:58:01 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:58:16 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506706.8359277,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506706.8555818,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506706.8556437,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:58:26 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:58:41 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506736.8369205,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506736.8489034,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506736.8489633,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:58:56 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:59:11 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:59:21 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506766.8378096,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506766.852042,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506766.8521018,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 06:59:31 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 06:59:46 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506796.8382816,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506796.8506846,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506796.8507414,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:00:01 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:00:11 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506826.838558,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506826.852035,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506826.8521013,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:00:26 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:00:36 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:00:46 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506856.8386526,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506856.8502524,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506856.850491,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:01:01 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:01:11 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506886.839704,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506886.852312,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506886.8523822,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:01:26 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:01:41 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506916.8405066,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506916.8531036,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506916.8531787,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:01:56 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:02:11 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:02:21 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506946.8414593,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506946.8531253,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506946.8533547,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:02:31 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:02:41 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:02:51 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781506976.8415296,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781506976.8533921,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781506976.8534565,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:03:01 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:03:11 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:03:21 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781507006.84207,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781507006.8536665,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781507006.8537338,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:03:36 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:03:51 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781507036.8425627,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781507036.8566005,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781507036.8566592,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:04:01 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:04:16 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781507066.843207,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781507066.8638272,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781507066.8638875,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:04:31 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:04:41 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781507096.843783,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781507096.857524,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781507096.8576005,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:04:56 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:05:11 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:05:21 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781507126.844621,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781507126.856495,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781507126.8565829,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:05:36 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:05:46 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781507156.845086,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781507156.8586278,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781507156.8586874,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:06:01 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:06:16 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781507186.8454757,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781507186.8575869,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781507186.8576422,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:06:26 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:06:36 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] 2026/06/15 07:06:51 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] {"level":"info","ts":1781507216.8460083,"logger":"allowlist-validator","msg":"InferencePool updated","name":"router-with-refs-pd-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1781507216.8581145,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] {"level":"info","ts":1781507216.8581693,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.134.0.48":{},"10.134.0.49":{},"router-with-refs-pd-test-kserve-6f78896447-wshh4":{},"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp":{}}} [e2e-llm-inference-service] 2026/06/15 07:07:01 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:52:27.310 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 06:52:27.310 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_438b1432-d4be-430d-9c68-f849a14f3ff0'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e6621a7a-5dee-42f9-9e77-bd97c022af0a'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_c1ec6ccb-f38d-47ae-a80c-c743eb6d07d0'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_c4b1a6dc-792d-4558-ac30-8d4d2f9c4b58'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/gPcsVCQDYDHk-_n0G9uADl7PXIM=.61c60ec52ed43038fff0fbbd68b080c94b0d94b4c8458dbd65965f9b17631c89.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_99ea61d6-edc4-43f8-80a7-3a5c0320c169'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_8a0e9d8b-b09b-4694-b7bd-35fd02930645'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_4c828383-b7b7-4ec0-9293-3a9e336f2a48'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Q1p2l2BzM1m6P5jKvr8WTq1TUio=.2d74da6615135c58cf3cf9ad4cb11e7c613ff9e55fe658a47ab83b6c8d1174a9.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ce31c37f-e910-4560-80ac-b06d461f8846'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_3601d903-51ed-48ad-8430-90c17768820e'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/a7eHxRFT3OeMBIFg52k2nfj5m7w=.db7090b0c8b34dd957a7e0656c718f978f9203cc874018f37dda44108be5970a.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_7b1642d8-08bc-4eb8-875a-5fd0dafc1766'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_8f1e6f6c-7e3f-4208-8865-4bdefba91c47'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_a473b17a-774f-47a4-9dfa-3d0fae70408d'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-06-15 06:52:31.529 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 06:52:31.529 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 4.2190895930002625 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 (EngineCore pid=70) DEBUG 06-15 06:53:45 [plugins/__init__.py:46] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:45 [plugins/__init__.py:46] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:45 [plugins/__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load. [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:45 [v1/engine/core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='/mnt/models', speculative_config=None, tokenizer='/mnt/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=facebook/opt-125m, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': True, 'scalar_asserts': True, 'dce': True, 'nan_asserts': False, 'epilogue_fusion': True, 'cpp.dynamic_threads': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'hidden_states', 'input_embeds'] [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['num_tokens_no_spec', 'token_ids_gpu', 'combined_mask'] [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [v1/worker/cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W615 06:53:46.151506535 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W615 06:53:46.151538343 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 70: [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [v1/worker/cpu_worker.py:109] OMP tid: 70, core 4 [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [v1/worker/cpu_worker.py:109] OMP tid: 87, core 5 [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [v1/worker/cpu_worker.py:109] OMP tid: 88, core 6 [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [v1/worker/cpu_worker.py:109] OMP tid: 89, core 7 [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [v1/worker/cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.134.0.48:57649 backend=gloo [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.134.0.48:57649 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors). [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading). [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:53:46 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [config/compilation.py:1194] enabled custom ops: Counter() [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [config/compilation.py:1195] disabled custom ops: Counter({'vocab_parallel_embedding': 1, 'logits_processor': 1}) [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:53:46 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ... [e2e-llm-inference-service] (EngineCore pid=70) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00 [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:53:55 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:05 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:15 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:25 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:54:27 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/86c9c3c579382eef68a98ac1d59b39811ba08abef3b4e90675a32c8dec3d7c90/rank_0_0/model [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:54:28 [compilation/monitor.py:76] Initial profiling/warmup run took 1.36 s [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:54:28 [v1/worker/cpu_model_runner.py:92] Warming up done. [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:54:28 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 41.86 seconds [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:54:28 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode='hf' [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:54:29 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1 [e2e-llm-inference-service] (EngineCore pid=70) INFO 06-15 06:54:29 [config/vllm.py:790] Asynchronous scheduling is disabled. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:29 [v1/engine/utils.py:1158] READY from local core engine process 0. [e2e-llm-inference-service] (EngineCore pid=70) WARNING 06-15 06:54:29 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:54:29 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:29 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:54:29 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 06-15 06:54:29 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:29 [entrypoints/openai/api_server.py:590] Supported tasks: ['generate'] [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:29 [renderers/base.py:197] Warming up chat template processing... [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:141] There is no chat template fallback for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [renderers/hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:30 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001 [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /docs, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /redoc, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:54:30 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:54:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:20 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:30 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:40 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:50 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:00 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:10 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:52:27.200 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 06:52:27.200 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_13077371-49cf-4e77-8654-0be4f72bc8b0'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_6b0d8e2a-6a5e-486d-9b3b-6bc2340c9e42'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_dac39dfa-8cd3-4c68-a30f-a9688409560f'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_28a4af8f-8c3d-4d18-ad1c-e0d75713c16f'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/gPcsVCQDYDHk-_n0G9uADl7PXIM=.61c60ec52ed43038fff0fbbd68b080c94b0d94b4c8458dbd65965f9b17631c89.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_2c75da64-24b6-4f3d-b447-4aeac8e05ee3'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_fe0f0baf-9547-4342-b956-4b82e7310b16'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_74804b8e-925f-4b23-9c06-26412d8d6d40'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Q1p2l2BzM1m6P5jKvr8WTq1TUio=.2d74da6615135c58cf3cf9ad4cb11e7c613ff9e55fe658a47ab83b6c8d1174a9.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e2b0889a-01a0-4c6a-9af2-616ba839dd57'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ca44ddec-8fcd-4da0-a3af-16e9bce90e2a'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/a7eHxRFT3OeMBIFg52k2nfj5m7w=.db7090b0c8b34dd957a7e0656c718f978f9203cc874018f37dda44108be5970a.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_bf287630-0009-475d-b9ba-9f1ec53be23d'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_83a53960-e2b6-402b-979d-57b050a7daef'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_30f78cb6-a425-4910-9d6c-b3098781fb3b'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-06-15 06:53:18.048 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 06:53:18.048 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 50.847708882999996 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 (APIServer pid=1) DEBUG 06-15 06:55:15 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8000 [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /docs, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /redoc, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:15 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 06-15 06:55:16 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:55:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:56:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:57:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:58:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 06:59:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:00:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:01:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:02:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:03:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:04:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 06:52:28.873 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 06:52:28.874 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-06-15 06:52:28.874 1 storage.initializer INFO [kserve_storage.py:download():169] Allow patterns: ['tokenizer.json', 'tokenizer_config.json', 'special_tokens_map.json', 'vocab.json', 'merges.txt', 'config.json', 'generation_config.json'] [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_14f05358-0ebb-4baa-b005-5c108c9af19c'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_789b3bd2-f20f-4701-806c-cbef077ce025'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_57acbcc4-eb2d-47b3-ac1e-fe3d6d8c94ce'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_f32abd54-67de-4d80-8297-7408a0abaa15'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_7a01b273-5be7-4c1a-b174-e177e102572a'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_5122ef55-2265-48b3-a31f-b17443c77313'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-06-15 06:52:29.324 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 06:52:29.324 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 0.45074305500020273 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"setup","caller":"runner/runner.go:150","msg":"GIE build","commit-sha":"","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"setup","caller":"runner/runner.go:169","msg":"Flags processed","flags":{"cache-info-metric":"vllm:cache_config_info","cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"kv-cache-usage-percentage-metric":"vllm:kv_cache_usage_perc","lora-info-metric":"vllm:lora_requests_info","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"router-with-refs-pd-test-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"total-queued-requests-metric":"vllm:num_requests_waiting","total-running-requests-metric":"vllm:num_requests_running","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"setup.trace","caller":"tracing/telemetry.go:131","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"loader/configloader.go:65","msg":"Loaded raw configuration","config":"{FeatureGates: {}, Plugins: [{/disagg-headers-handler} {/prefill-filter} {/decode-filter} {/queue-scorer} {/prefix-cache-scorer} {/max-score-picker} {/always-disagg-pd-decider} {/disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"prefix/plugin.go:203","msg":"BlockSize is not positive, using default value","default":16} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"prefix/plugin.go:213","msg":"PrefixCachePlugin initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":256,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"profile/disagg_profile_handler.go:168","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"loader/configloader.go:98","msg":"Effective configuration loaded","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","kind":"EndpointPickerConfig"},"configError":"got runtime.Object without object metadata: {FeatureGates: {}, Plugins: [{disagg-headers-handler/disagg-headers-handler} {prefill-filter/prefill-filter} {decode-filter/decode-filter} {queue-scorer/queue-scorer} {prefix-cache-scorer/prefix-cache-scorer} {max-score-picker/max-score-picker} {always-disagg-pd-decider/always-disagg-pd-decider} {disagg-profile-handler/disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {fcfs-ordering-policy/fcfs-ordering-policy} {global-strict-fairness-policy/global-strict-fairness-policy}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"runner/runner.go:549","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"setup","caller":"runner/runner.go:301","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"setup","caller":"runner/runner.go:315","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","logger":"setup.SaturationDetector","caller":"utilizationdetector/detector.go:70","msg":"Creating new SaturationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"setup","caller":"runner/runner.go:350","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"setup","caller":"runner/runner.go:644","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"setup","caller":"runner/runner.go:209","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T06:52:30Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:52:30Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"router-with-refs-pd-test-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"router-with-refs-pd-test-inference-pool","reconcileID":"aaa01c34-b825-4635-a9d1-c921a983162d","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:54:36Z","caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"router-with-refs-pd-test-kserve-6f78896447-wshh4","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"router-with-refs-pd-test-kserve-6f78896447-wshh4","reconcileID":"f859bdef-5385-40db-ae03-c1c892ab478a"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:54:36Z","caller":"metrics/pod_metrics.go:76","msg":"Starting refresher","endpoint":{"name":"router-with-refs-pd-test-kserve-6f78896447-wshh4-rank-0","namespace":"kserve-ci-e2e-test"},"metadata":"{NamespacedName:kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4-rank-0 PodName:router-with-refs-pd-test-kserve-6f78896447-wshh4 Address:10.134.0.48 Port:8000 MetricsHost:10.134.0.48:8000 Labels:map[app.kubernetes.io/component:llminferenceservice-workload app.kubernetes.io/name:router-with-refs-pd-test app.kubernetes.io/part-of:llminferenceservice kserve.io/component:workload llm-d.ai/role:decode pod-template-hash:6f78896447]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:55:16Z","caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp","reconcileID":"816353c7-c530-4478-b667-03103138f70d"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T06:55:16Z","caller":"metrics/pod_metrics.go:76","msg":"Starting refresher","endpoint":{"name":"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp-rank-0","namespace":"kserve-ci-e2e-test"},"metadata":"{NamespacedName:kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp-rank-0 PodName:router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp Address:10.134.0.49 Port:8000 MetricsHost:10.134.0.49:8000 Labels:map[app.kubernetes.io/component:llminferenceservice-workload-prefill app.kubernetes.io/name:router-with-refs-pd-test app.kubernetes.io/part-of:llminferenceservice kserve.io/component:workload llm-d.ai/role:prefill pod-template-hash:5fc8578dd5]}"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'tokenizer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 INFO 06-15 06:52:35 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 06-15 06:52:35 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] 2026-06-15 06:52:37,541 [INFO] [root] TokenizationServiceServicer initialized [e2e-llm-inference-service] 2026-06-15 06:52:37,541 [INFO] [root] gRPC reflection disabled (set `ENABLE_GRPC_REFLECTION=1` to enable) [e2e-llm-inference-service] 2026-06-15 06:52:37,541 [INFO] [root] gRPC server configured to listen on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:52:37,541 [INFO] [root] gRPC server started on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 06:52:37,542 [INFO] [root] Probe server started on port 8082 [e2e-llm-inference-service] 2026-06-15 06:52:37,542 [INFO] [root] Server started. [e2e-llm-inference-service] 2026-06-15 06:52:38,349 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:38 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:43 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:58,348 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:52:59,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:52:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:09,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:28 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:29,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:53:59,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:53:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:19 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:54:59,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:54:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:19,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:19 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:29 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:39,167 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:43,346 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:43 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:55:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:55:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:39 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:49 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:56:59,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:56:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:29 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:39 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:43 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:57:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:57:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:09 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:43,346 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:58 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:58:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:58:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:13,346 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 06:59:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:06:59:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:39,167 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:43 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:19,167 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:19 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:29 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:39,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:49 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:58 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:13 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:39 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:43 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:29,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:09 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:28,346 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:29,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:29 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:59,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:59 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:19 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:28,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:39 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:49 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:58,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:09,165 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:09 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:13 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:19,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:19 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:28,346 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:28 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:29,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:29 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:39,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:39 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:43,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:43 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:49,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:49 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:58,346 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:58 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:59,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:59 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:09,166 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:09 +0000] "GET /healthz HTTP/1.1" 200 261 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:13,347 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:13 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e41c917e-47f0-4efd-9153-79fffe84bad0 [e2e-llm-inference-service] resourceVersion: '60856' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.45 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b [e2e-llm-inference-service] uid: bb6c90a3-f29e-457c-b4ff-2d9c233c7b18 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b6dd44b1-3f7b-4b81-b0a1-8724d0eae99e [e2e-llm-inference-service] resourceVersion: '63043' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.48 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-6f78896447-wshh4 [e2e-llm-inference-service] uid: 5d2cc4b0-4097-4d57-9116-4a62c41cc76d [e2e-llm-inference-service] - ip: 10.134.0.49 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp [e2e-llm-inference-service] uid: 9c26092a-3ad8-4fc2-879a-4b96a47c9166 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-6f78896447-wshh4 [e2e-llm-inference-service] generateName: router-with-refs-pd-test-kserve-6f78896447- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5d2cc4b0-4097-4d57-9116-4a62c41cc76d [e2e-llm-inference-service] resourceVersion: '62193' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6f78896447 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.48/23"],"mac_address":"0a:58:0a:86:00:30","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.48/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.48\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:30\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-6f78896447 [e2e-llm-inference-service] uid: 3de390c7-64f1-433b-af14-e1d9f47e1412 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-226 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3de390c7-64f1-433b-af14-e1d9f47e1412"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:54:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.48"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-s2zn9 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-s2zn9 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-s2zn9 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n ucx_hcas=()\n for hca in \"${active_hcas[@]}\"\ [e2e-llm-inference-service] ; do\n ucx_hcas+=(\"${hca}:1\")\n done\n\n # Check if we found any active\ [e2e-llm-inference-service] \ HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements\ [e2e-llm-inference-service] \ with a comma\n hcas=$(IFS=,; echo \"${active_hcas[*]}\")\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Setting active HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n\ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if counts are equal - prefer lower index number\n\ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8001\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-s2zn9 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-pd-test-kserve [e2e-llm-inference-service] serviceAccount: router-with-refs-pd-test-kserve [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: router-with-refs-pd-test-kserve-dockercfg-nt6r8 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:31Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:54:36Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:54:36Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] hostIP: 10.0.128.226 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.226 [e2e-llm-inference-service] podIP: 10.134.0.48 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.48 [e2e-llm-inference-service] startTime: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-routing-sidecar@sha256:14ff2530c83bd6f95fa5b25309b150623b403da83f9152f635858f02163e2f95 [e2e-llm-inference-service] containerID: cri-o://2c5583e9d57e610617e4afe818b762c966b02171e8da1bb9cb31b6b6c39db1bc [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-s2zn9 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T06:52:31Z' [e2e-llm-inference-service] containerID: cri-o://05b64457d70b2f1f5aced8df47a0ac28d696157ad85c665ae991a14efd8d41c3 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://05b64457d70b2f1f5aced8df47a0ac28d696157ad85c665ae991a14efd8d41c3 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-s2zn9 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:52:32Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0 [e2e-llm-inference-service] containerID: cri-o://d68822c89ab4c22d8c6af65fd68d08df9f1c852252b8d75890f4192af6f26a69 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-s2zn9 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp [e2e-llm-inference-service] generateName: router-with-refs-pd-test-kserve-prefill-5fc8578dd5- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9c26092a-3ad8-4fc2-879a-4b96a47c9166 [e2e-llm-inference-service] resourceVersion: '63041' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 5fc8578dd5 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.49/23"],"mac_address":"0a:58:0a:86:00:31","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.49/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.49\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:31\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-prefill-5fc8578dd5 [e2e-llm-inference-service] uid: b26d0305-15ed-46d8-987e-75acf4c15489 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-226 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b26d0305-15ed-46d8-987e-75acf4c15489"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.49"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-kfcvt [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-kfcvt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n ucx_hcas=()\n for hca in \"${active_hcas[@]}\"\ [e2e-llm-inference-service] ; do\n ucx_hcas+=(\"${hca}:1\")\n done\n\n # Check if we found any active\ [e2e-llm-inference-service] \ HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements\ [e2e-llm-inference-service] \ with a comma\n hcas=$(IFS=,; echo \"${active_hcas[*]}\")\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Setting active HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n\ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if counts are equal - prefer lower index number\n\ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-kfcvt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:53:18Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] hostIP: 10.0.128.226 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.226 [e2e-llm-inference-service] podIP: 10.134.0.49 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.49 [e2e-llm-inference-service] startTime: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T06:53:18Z' [e2e-llm-inference-service] containerID: cri-o://b8415f513b0225674e4ba57b4ebab3ffddda0b02f5a7f22ff52ec88b94a4e1ef [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://b8415f513b0225674e4ba57b4ebab3ffddda0b02f5a7f22ff52ec88b94a4e1ef [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-kfcvt [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:53:18Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0 [e2e-llm-inference-service] containerID: cri-o://fe63a26a8ca996b9f2ad0eb8a01417e5eee30c0abbe5167f727442ff8d283ce7 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-kfcvt [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b [e2e-llm-inference-service] generateName: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfb- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: bb6c90a3-f29e-457c-b4ff-2d9c233c7b18 [e2e-llm-inference-service] resourceVersion: '60854' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f7487fdfb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.45/23"],"mac_address":"0a:58:0a:85:00:2d","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.45/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.45\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:2d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfb [e2e-llm-inference-service] uid: 0c8740dd-1b7a-4125-9d09-4e1cc9c1ff89 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-25 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0c8740dd-1b7a-4125-9d09-4e1cc9c1ff89"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.45"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-t9kqp [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-t9kqp [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-t9kqp [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] - name: kube-api-access-t9kqp [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] serviceAccount: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: router-with-refs-pd-test-epp-sa-dockercfg-zqkmt [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:30Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] hostIP: 10.0.141.25 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.25 [e2e-llm-inference-service] podIP: 10.133.0.45 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.45 [e2e-llm-inference-service] startTime: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] containerID: cri-o://b0165dd19df93ec101af23e0273c86d22b0d98fc86065345f99a625f3d9da714 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://b0165dd19df93ec101af23e0273c86d22b0d98fc86065345f99a625f3d9da714 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-t9kqp [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:52:30Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-scheduler@sha256:88de279c6eb6758a4c600de9730e49e46b04c392846afedd03d82447379c9e7a [e2e-llm-inference-service] containerID: cri-o://416e6aa380be17d437d8fbc3bd0d12992e4125801d118845cc8706877afb3eb9 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-t9kqp [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T06:52:30Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-uds-tokenizer@sha256:aed091a51f3d64458f1fdb451d21f745186bb4517a7ba0c49913a0c617366a3e [e2e-llm-inference-service] containerID: cri-o://f4ef13056d3d95581c0612908e5ac35deeb001f9dec7e33f6c7404acd1e899f5 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-t9kqp [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3d59a50e-2033-481f-a64e-d870ce3652f5 [e2e-llm-inference-service] resourceVersion: '60288' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: router-with-refs-pd-test-epp-sa-dockercfg-zqkmt [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"router-with-refs-pd-test-epp-sa-dockercfg-zqkmt"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjfwp"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: router-with-refs-pd-test-epp-sa-dockercfg-zqkmt [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: router-with-refs-pd-test-epp-sa-dockercfg-zqkmt [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 385cab14-62a9-4839-9be5-bd584968dab5 [e2e-llm-inference-service] resourceVersion: '60235' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:25Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: router-with-refs-pd-test-kserve-dockercfg-nt6r8 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"router-with-refs-pd-test-kserve-dockercfg-nt6r8"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjfwp"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: router-with-refs-pd-test-kserve-dockercfg-nt6r8 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: router-with-refs-pd-test-kserve-dockercfg-nt6r8 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: de22b761-993e-4e78-afde-7276c7cb29bc [e2e-llm-inference-service] resourceVersion: '60363' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.30.8 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.30.8 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 00cb09b8-235c-4554-b834-eac448aa72d7 [e2e-llm-inference-service] resourceVersion: '60268' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.198.186 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.198.186 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 90b48961-4b3b-49cb-84c9-f28da175b27c [e2e-llm-inference-service] resourceVersion: '62197' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:54:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-pd-test-kserve [e2e-llm-inference-service] serviceAccount: router-with-refs-pd-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:54:36Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:54:36Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:54:36Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "router-with-refs-pd-test-kserve-6f78896447" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 726fb5c3-df02-4357-a03d-bfe9e0ab9d54 [e2e-llm-inference-service] resourceVersion: '63045' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "router-with-refs-pd-test-kserve-prefill-5fc8578dd5" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 913912b8-37a0-416c-a402-6894467656a9 [e2e-llm-inference-service] resourceVersion: '60859' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type:\ [e2e-llm-inference-service] \ max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] serviceAccount: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfb" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-6f78896447 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3de390c7-64f1-433b-af14-e1d9f47e1412 [e2e-llm-inference-service] resourceVersion: '62196' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6f78896447 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve [e2e-llm-inference-service] uid: 90b48961-4b3b-49cb-84c9-f28da175b27c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"90b48961-4b3b-49cb-84c9-f28da175b27c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:54:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6f78896447 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6f78896447 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-pd-test-kserve [e2e-llm-inference-service] serviceAccount: router-with-refs-pd-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-prefill-5fc8578dd5 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b26d0305-15ed-46d8-987e-75acf4c15489 [e2e-llm-inference-service] resourceVersion: '63044' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 5fc8578dd5 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] uid: 726fb5c3-df02-4357-a03d-bfe9e0ab9d54 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"726fb5c3-df02-4357-a03d-bfe9e0ab9d54"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 5fc8578dd5 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 5fc8578dd5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0c8740dd-1b7a-4125-9d09-4e1cc9c1ff89 [e2e-llm-inference-service] resourceVersion: '60858' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f7487fdfb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 913912b8-37a0-416c-a402-6894467656a9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"913912b8-37a0-416c-a402-6894467656a9"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f7487fdfb [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f7487fdfb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type:\ [e2e-llm-inference-service] \ max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] serviceAccount: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 149982cf-8798-4a81-87b9-d12202ab4ae0 [e2e-llm-inference-service] resourceVersion: '60327' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f3b5d1d5-eb63-4cb6-bec9-797c10eadb9b [e2e-llm-inference-service] resourceVersion: '60246' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0f590030-d02f-4a0a-b4b8-9ff82c1f0798 [e2e-llm-inference-service] resourceVersion: '60309' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ee8768eb-40bd-4dae-af90-8e71d36c7083 [e2e-llm-inference-service] resourceVersion: '60243' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-service-nhgdp [e2e-llm-inference-service] generateName: router-with-refs-pd-test-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7a450533-bfa1-450f-bec2-03884757589f [e2e-llm-inference-service] resourceVersion: '60857' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: router-with-refs-pd-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-service [e2e-llm-inference-service] uid: de22b761-993e-4e78-afde-7276c7cb29bc [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:53:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"de22b761-993e-4e78-afde-7276c7cb29bc"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.45 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b [e2e-llm-inference-service] uid: bb6c90a3-f29e-457c-b4ff-2d9c233c7b18 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-workload-svc-pgfbl [e2e-llm-inference-service] generateName: router-with-refs-pd-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f1ce30db-b42f-4239-93dd-04e0a75e1238 [e2e-llm-inference-service] resourceVersion: '63042' [e2e-llm-inference-service] generation: 5 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] uid: 00cb09b8-235c-4554-b834-eac448aa72d7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:55:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"00cb09b8-235c-4554-b834-eac448aa72d7"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.49 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp [e2e-llm-inference-service] uid: 9c26092a-3ad8-4fc2-879a-4b96a47c9166 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.48 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-6f78896447-wshh4 [e2e-llm-inference-service] uid: 5d2cc4b0-4097-4d57-9116-4a62c41cc76d [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 149982cf-8798-4a81-87b9-d12202ab4ae0 [e2e-llm-inference-service] resourceVersion: '60327' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f3b5d1d5-eb63-4cb6-bec9-797c10eadb9b [e2e-llm-inference-service] resourceVersion: '60246' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:router-with-refs-pd-test-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0f590030-d02f-4a0a-b4b8-9ff82c1f0798 [e2e-llm-inference-service] resourceVersion: '60309' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ee8768eb-40bd-4dae-af90-8e71d36c7083 [e2e-llm-inference-service] resourceVersion: '60243' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:52:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] name: router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60377' [e2e-llm-inference-service] uid: 83ffd49f-f9fe-4dae-b04b-b6c3c0109e67 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: {} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:14Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:52:17Z' [e2e-llm-inference-service] name: router-route-3-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60106' [e2e-llm-inference-service] uid: 1db8639b-cf2a-4c7f-a6ec-a5a310653b45 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: router-route-3 [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:15Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:17Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:14Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T06:52:17Z' [e2e-llm-inference-service] name: router-route-4-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60104' [e2e-llm-inference-service] uid: f2aee398-f8be-4d32-b178-b3a3bf5a2784 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: router-route-4 [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:15Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:52:17Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60396' [e2e-llm-inference-service] uid: 0bf7ced3-ac46-4798-8775-ac6b60f43d63 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-pd-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-pd-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60402' [e2e-llm-inference-service] uid: bd7c023c-8e77-44ee-8f80-d1f6285aa925 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-pd-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-pd-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60396' [e2e-llm-inference-service] uid: 0bf7ced3-ac46-4798-8775-ac6b60f43d63 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-pd-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-pd-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60402' [e2e-llm-inference-service] uid: bd7c023c-8e77-44ee-8f80-d1f6285aa925 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-pd-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-pd-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60396' [e2e-llm-inference-service] uid: 0bf7ced3-ac46-4798-8775-ac6b60f43d63 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-pd-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-pd-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:29Z' [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60402' [e2e-llm-inference-service] uid: bd7c023c-8e77-44ee-8f80-d1f6285aa925 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: router-with-refs-pd-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: router-with-refs-pd-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c8d5cfc-ec66-4796-9dba-9a636e8753f7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:52:28Z' [e2e-llm-inference-service] name: router-with-refs-pd-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: router-with-refs-pd-test [e2e-llm-inference-service] uid: 2c8d5cfc-ec66-4796-9dba-9a636e8753f7 [e2e-llm-inference-service] resourceVersion: '60380' [e2e-llm-inference-service] uid: ba68bf84-3513-4394-a594-7d0489aecad4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: router-with-refs-pd-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-6f78896447-wshh4 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:07:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6f78896447 [e2e-llm-inference-service] timestamp: '2026-06-15T07:07:06Z' [e2e-llm-inference-service] window: 22.775s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 16664061n [e2e-llm-inference-service] memory: 20996Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 107984099n [e2e-llm-inference-service] memory: 2407604Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:07:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 5fc8578dd5 [e2e-llm-inference-service] timestamp: '2026-06-15T07:07:01Z' [e2e-llm-inference-service] window: 17.117s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 108556230n [e2e-llm-inference-service] memory: 2398256Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:07:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: router-with-refs-pd-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f7487fdfb [e2e-llm-inference-service] timestamp: '2026-06-15T07:07:03Z' [e2e-llm-inference-service] window: 13.291s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 234594n [e2e-llm-inference-service] memory: 360020Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 123163981n [e2e-llm-inference-service] memory: 29656Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-06-15T07:07:17.200751] end - ❌ 903.013s: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/router-with-refs-pd-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T06:54:37Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T06:52:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/router-route-3: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T06:53:00Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T06:55:17Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_n... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:727: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'workload-sin... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-06-15T07:01:51.337397', start_time = 1781506911.3376548 [e2e-llm-inference-service] duration = 900.2222678661346, timestamp_end = '2026-06-15T07:16:51.559924' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_n... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] extra_headers = {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'} [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1030: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f7425e9bec0> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 401: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1026: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1445 Checking LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1471 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1481 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-06-15T06:59:42.525730] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_name='llmisvc-model-fb-opt-125m-with-ba4d693a', endpoint='/v1/models', max_tokens=20, payload_formatter=None, response_assertion=.response_assertion at 0x7f7426d48040>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1496 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-06-15T06:59:42.542056] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-06-15T06:59:42.637151] end - ✅ in 0.095s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-06-15T06:59:42.637236] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T07:00:22Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T07:00:22Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T07:00:22Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T07:00:22Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T07:00:22Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-06-15T07:00:22Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-06-15T07:00:22Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-06-15T07:00:22Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-15T07:00:31Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-06-15T07:00:31Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-06-15T07:00:02Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-06-15T07:01:51.337261] end - ✅ in 128.700s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-06-15T07:01:51.337397] start - args=(, TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_name='llmisvc-model-fb-opt-125m-with-ba4d693a', endpoint='/v1/models', max_tokens=20, payload_formatter=None, response_assertion=.response_assertion at 0x7f7426d48040>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:01:51.337659] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:01:51.345839] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1133 Waiting: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:01:56.453660] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:01:56.462808] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:01.538261] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:01.547580] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:06.585098] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:06.593872] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:11.631788] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:11.639990] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:16.670955] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:16.679537] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:21.704594] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:21.714346] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:26.747606] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:26.756538] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:31.781703] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:31.790935] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:36.814309] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:36.823482] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:41.852148] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:41.860739] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:46.893332] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:46.906444] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:51.932817] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:51.941531] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:02:56.964970] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:02:56.976475] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:02.134292] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:02.143105] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:07.170657] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:07.180035] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:12.208556] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:12.217967] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:17.246773] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:17.256267] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:22.283873] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:22.293714] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:27.442714] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:27.451205] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:32.477258] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:32.486288] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:37.514793] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:37.524772] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:42.553006] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:42.561789] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:47.593718] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:47.608049] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:52.633395] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:52.643265] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:03:57.670908] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:03:57.681465] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:02.710752] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:02.720638] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:07.751439] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:07.764818] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:12.798516] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:12.819646] end - ✅ in 0.020s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:17.849079] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:17.860289] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:22.956493] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:22.966171] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:27.996807] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:28.013662] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:33.042250] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:33.055887] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:38.086112] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:38.097127] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:43.128220] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:43.140702] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:48.162542] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:48.176527] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:53.205522] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:53.230854] end - ✅ in 0.025s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:04:58.354097] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:04:58.425223] end - ✅ in 0.070s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:03.453218] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:03.465820] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:08.496370] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:08.516801] end - ✅ in 0.020s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:13.645159] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:13.667161] end - ✅ in 0.022s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:18.691339] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:18.734746] end - ✅ in 0.043s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:23.765961] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:23.775083] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:28.803697] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:28.813736] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:33.842016] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:33.851330] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:38.882372] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:38.892091] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:43.917950] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:43.926734] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:48.959762] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:48.969451] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:54.054506] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:54.063114] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:05:59.159251] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:05:59.175284] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:04.213872] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:04.230047] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:09.336141] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:09.345774] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:14.376328] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:14.385298] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:19.414891] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:19.424647] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:24.449720] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:24.461236] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:29.596547] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:29.606078] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:34.637882] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:34.648168] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:39.692853] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:39.702113] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:44.729275] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:44.741021] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:49.772132] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:49.781798] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:54.809203] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:54.818162] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:06:59.953282] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:06:59.962202] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:04.994827] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:05.003562] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:10.036438] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:10.045829] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:15.075329] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:15.085087] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:20.112019] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:20.187009] end - ✅ in 0.074s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:25.213974] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:25.223167] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:30.244586] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:30.253711] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:35.295381] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:35.488762] end - ✅ in 0.193s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:40.521452] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:40.587473] end - ✅ in 0.065s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:45.620344] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:45.687185] end - ✅ in 0.066s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:50.723187] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:50.733523] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:07:55.764406] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:07:55.773797] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:00.814187] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:00.822944] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:05.904856] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:05.914335] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:10.944902] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:10.954407] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:15.986493] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:15.995558] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:21.024289] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:21.037135] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:26.196798] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:26.205574] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:31.231071] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:31.240114] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:36.268030] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:36.277041] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:41.310799] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:41.319752] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:46.344423] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:46.355240] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:51.384792] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:51.394838] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:08:56.425639] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:08:56.435361] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:01.476266] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:01.486665] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:06.522086] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:06.531118] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:11.627278] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:11.636378] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:16.667498] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:16.677059] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:21.711439] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:21.721181] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:26.748666] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:26.759049] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:31.788613] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:31.798451] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:36.825976] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:36.838622] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:41.870139] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:41.879496] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:46.915414] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:46.924865] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:51.952303] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:51.962247] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:09:56.994428] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:09:57.004063] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:02.030450] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:02.040592] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:07.070343] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:07.079815] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:12.109377] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:12.118773] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:17.141526] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:17.151662] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:22.193015] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:22.204715] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:27.227118] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:27.237124] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:32.260582] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:32.388376] end - ✅ in 0.127s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:37.422330] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:37.488043] end - ✅ in 0.065s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:42.518712] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:42.528309] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:47.598247] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:47.609581] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:52.644817] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:52.687212] end - ✅ in 0.042s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:10:57.717339] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:10:57.887814] end - ✅ in 0.170s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:02.917295] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:02.987794] end - ✅ in 0.070s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:08.021092] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:08.032479] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:13.117872] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:13.128379] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:18.159596] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:18.169946] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:23.200606] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:23.210704] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:28.250096] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:28.259951] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:33.289041] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:33.298529] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:38.328778] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:38.342246] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:43.364915] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:43.375135] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:48.411348] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:48.423472] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:53.452594] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:53.462200] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:11:58.493038] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:11:58.503195] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:03.598410] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:03.609348] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:08.644300] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:08.653858] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:13.683137] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:13.693203] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:18.728889] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:18.738505] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:23.773878] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:23.788815] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:28.990995] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:29.007755] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:34.039979] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:34.049079] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:39.102128] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:39.111297] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:44.136107] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:44.146370] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:49.186303] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:49.195041] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:54.227123] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:54.237373] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:12:59.265630] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:12:59.275814] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:04.303800] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:04.313509] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:09.342270] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:09.352502] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:14.383820] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:14.393395] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:19.424852] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:19.433950] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:24.465514] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:24.475136] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:29.505265] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:29.516337] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:34.543497] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:34.553169] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:39.579741] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:39.589275] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:44.620429] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:44.629200] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:49.654340] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:49.663221] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:54.692888] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:54.702528] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:13:59.730996] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:13:59.741754] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:04.771874] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:04.781997] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:09.811142] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:09.819905] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:14.853406] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:14.862212] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:19.890360] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:19.899539] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:24.929538] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:24.940993] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:29.969167] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:29.978713] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:35.014548] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:35.023601] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:40.052254] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:40.062361] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:45.090615] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:45.100313] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:50.140353] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:50.149452] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:14:55.175709] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:14:55.185955] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:00.216327] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:00.226081] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:05.256847] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:05.267004] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:10.292349] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:10.301749] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:15.327527] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:15.337929] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:20.367951] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:20.377155] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:25.405638] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:25.415592] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:30.442571] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:30.451971] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:35.478426] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:35.487622] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:40.514002] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:40.524157] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:45.694972] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:45.705159] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:50.824942] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:50.834179] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:15:55.860830] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:15:55.869345] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:00.892842] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:00.902135] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:05.932712] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:05.942312] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:10.972617] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:10.981652] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:16.012772] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:16.021918] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:21.051330] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:21.060001] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:26.087811] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:26.097298] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:31.198881] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:31.209647] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:36.401560] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:36.410770] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:41.441082] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:41.452392] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:46.484517] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:46.494534] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-06-15T07:16:51.521487] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:180 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-06-15T07:16:51.531388] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1003 Calling LLM service at http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1022 Model response is 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1130 Timed out waiting: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-06-15T07:16:51.559924] end - ❌ 900.222s: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:742 [router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf] ❌ ERROR: Failed to call llm inference service llmisvc-model-fb-opt-125m-with-ba4d693a: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1151 🔍 # Diagnostics for 'llmisvc-model-fb-opt-125m-with-ba4d693a' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1152 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1153 # LLMInferenceService llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1156 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:42Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:59:42Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:59:42Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T07:01:50Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '68008' [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] - name: workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] - name: model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:22Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:22Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:01:50Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:02Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:01:50Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:01:50Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a2e817a52d48e474fb69337e00bd8658-600493333.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-699694bb49-m6gc4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-b5799d8f5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-699694bb49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:21:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:23:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-85d86d876c-vrqhw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" in 3.371s (3.371s including waiting). Image size: 299992506 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-85d86d876c-vrqhw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6c5d597fbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-85d86d876c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.36s (27.36s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:49 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-f5744d7b7-gjb94 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-7748b48dbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-f5744d7b7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:20:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.46:8001/health": dial tcp 10.134.0.46:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6b5b6588r7 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-6b5b6588r7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-6b5b695dd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-78b45dc7ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:53 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:49:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:25 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:55 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.39:8000/health": dial tcp 10.134.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-598d8c75cc-qw9md [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-54bd696fdf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-598d8c75cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-779977f94cgh8n6 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-779977f94c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.286s (1.286s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:09:44 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.51:8000/health": dial tcp 10.134.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:09:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:19 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.51:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:27 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:09:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" in 867ms (867ms including waiting). Image size: 67767940 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.44:8001/health": dial tcp 10.134.0.44:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8649d9d4d8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:09 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:56 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:50 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:07 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:34 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" in 3.932s (3.932s including waiting). Image size: 299992506 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:34 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:34 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:45 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:11:13 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.461s (27.461s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:11:13 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:11:13 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:37 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:12:10 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.53:8000/health": dial tcp 10.134.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 588ff956b for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:12:20 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd44w7x4 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:35 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:20 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-76859b4cd4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-7cdd64995b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:14 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:15 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:16 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:16 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:00 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:01:39 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.50:8000/health": dial tcp 10.134.0.50:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-86f69d9999 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:00 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:01 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:59:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:00:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:26:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:07:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c-g56wd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-598bf9c8c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:29 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-7f6b44d55d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:30 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:31 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:31 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 07:10:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6jhfg to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:22 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:24 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:48:27 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6jhfg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:45:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-57bd5888f4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-7bc88f48bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqcskz to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:11 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-548bd48954 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:57 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:24:58 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:27:11 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-5597d7fd6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:25:42 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:06:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-68b6785c7d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-67h82 to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.023s (1.023s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:44 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-67h82 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5c556785f6-h6wcn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-74dcd66dl8zmb to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:38 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:57 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.38:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-74dcd66d7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5c556785f6 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:05:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:45 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.707s (2.707s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:27 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:33 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.28:15021/healthz/ready": dial tcp 10.134.0.28:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-75dcfd69c9-dh6qf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-75dcfd69c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:44 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:48 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:29:59 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-78c98f6f4c-ddrqp to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:12 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:14 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" in 2.491s (2.491s including waiting). Image size: 179625600 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:40be785b9abecd641f3121855a066c0ea01aba66e1350f33d175f2351c54e371" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-78c98f6f4c-ddrqp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-78c98f6f4c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:15 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:11 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:31 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:32 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:54:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.48:8001/health": dial tcp 10.134.0.48:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-6f78896447-wshh4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:53:18 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:55:06 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-5fc8578dd5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-6f78896447 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:52:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:24 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:32:12 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-578d595fc-gtvkx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:14 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:15 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7d4868d689 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-578d595fc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:30:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-96f8b89cb-j7r99 to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-96f8b89cb-j7r99 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-9c4tp6hx to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:32 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-9c4c7855f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-96f8b89cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:04:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq to ip-10-0-128-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.082s (1.082s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5d7479f884-f4vfq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf to ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 951ms (951ms including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 30.592s (30.592s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:22 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-128-226.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884sbpts to ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1" in 1.034s (1.034s including waiting). Image size: 87241021 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:51 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1" in 31.996s (31.996s including waiting). Image size: 2907020949 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:23 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:30 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.34:8082/healthz": dial tcp 10.133.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:08 kserve-ci-e2e-test kubelet/ip-10-0-141-25.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884fbb from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5d7479f884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:02:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-06-15 06:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 07:00:00.342 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models'), ('hf://edbeeching/opt-125m-lora', '/mnt/lora/lora-adapter-1')] [e2e-llm-inference-service] 2026-06-15 07:00:00.342 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-06-15 07:00:20.608 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 07:00:20.609 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 20.266322774999935 seconds. [e2e-llm-inference-service] 2026-06-15 07:00:20.609 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://edbeeching/opt-125m-lora to local [e2e-llm-inference-service] 2026-06-15 07:00:24.371 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://edbeeching/opt-125m-lora to /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] 2026-06-15 07:00:24.371 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 3.7627972380005303 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 (APIServer pid=1) DEBUG 06-15 07:05:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:05:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:06:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:07:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:08:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:09:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:10:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:11:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:12:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:13:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:14:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:53 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:15:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:03 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:13 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:23 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:33 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:43 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 06-15 07:16:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-06-15 07:00:00.608 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-06-15 07:00:00.609 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-06-15 07:00:00.609 1 storage.initializer INFO [kserve_storage.py:download():169] Allow patterns: ['tokenizer.json', 'tokenizer_config.json', 'special_tokens_map.json', 'vocab.json', 'merges.txt', 'config.json', 'generation_config.json'] [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_34a4774a-10f1-409b-8d5f-f0465589fe78'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_a9ebbdfb-7baf-4f18-ab27-e32d4b805aaf'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_de1a748c-96de-4b36-8b24-c119c3012028'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_b19d1cf8-9466-4be1-9ec9-2769f420d613'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_20c0ff76-695c-4076-9b26-bcfcba396f44'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_18c97928-d7a1-4fe3-8c09-624b66d9a220'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-06-15 07:00:01.101 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-06-15 07:00:01.101 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 0.49294182500034367 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"setup","caller":"runner/runner.go:150","msg":"GIE build","commit-sha":"","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"setup","caller":"runner/runner.go:169","msg":"Flags processed","flags":{"cache-info-metric":"vllm:cache_config_info","cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"kv-cache-usage-percentage-metric":"vllm:kv_cache_usage_perc","lora-info-metric":"vllm:lora_requests_info","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"total-queued-requests-metric":"vllm:num_requests_waiting","total-running-requests-metric":"vllm:num_requests_running","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"setup.trace","caller":"tracing/telemetry.go:131","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"loader/configloader.go:65","msg":"Loaded raw configuration","config":"{FeatureGates: {}, Plugins: [{/single-profile-handler} {/queue-scorer} {/prefix-cache-scorer} {/max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"prefix/plugin.go:203","msg":"BlockSize is not positive, using default value","default":16} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"prefix/plugin.go:213","msg":"PrefixCachePlugin initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":256,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"loader/configloader.go:98","msg":"Effective configuration loaded","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","kind":"EndpointPickerConfig"},"configError":"got runtime.Object without object metadata: {FeatureGates: {}, Plugins: [{single-profile-handler/single-profile-handler} {queue-scorer/queue-scorer} {prefix-cache-scorer/prefix-cache-scorer} {max-score-picker/max-score-picker} {fcfs-ordering-policy/fcfs-ordering-policy} {global-strict-fairness-policy/global-strict-fairness-policy}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.000000} {PluginRef: prefix-cache-scorer, Weight: 3.000000} {PluginRef: max-score-picker}]}], Data: , SaturationDetector: {}, FlowControl: }"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"runner/runner.go:549","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"setup","caller":"runner/runner.go:301","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"setup","caller":"runner/runner.go:315","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","logger":"setup.SaturationDetector","caller":"utilizationdetector/detector.go:70","msg":"Creating new SaturationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"setup","caller":"runner/runner.go:350","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"setup","caller":"runner/runner.go:644","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"setup","caller":"runner/runner.go:209","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:01Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","reconcileID":"8c07552d-b523-4f62-9141-8c9ff9a65516","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":"2026-06-15T07:00:01Z","caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:00:20Z","caller":"controller/inferencepool_reconciler.go:50","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","reconcileID":"1803ae55-7036-4653-a6f0-a4c556151294","group":"inference.networking.k8s.io"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:01:49Z","caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8","reconcileID":"7a0b2137-4c1f-4689-bbb9-32bdcb47c92a"} [e2e-llm-inference-service] {"level":"Level(-2)","ts":"2026-06-15T07:01:49Z","caller":"metrics/pod_metrics.go:76","msg":"Starting refresher","endpoint":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8-rank-0","namespace":"kserve-ci-e2e-test"},"metadata":"{NamespacedName:kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8-rank-0 PodName:llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 Address:10.134.0.50 Port:8000 MetricsHost:10.134.0.50:8000 Labels:map[app.kubernetes.io/component:llminferenceservice-workload app.kubernetes.io/name:llmisvc-model-fb-opt-125m-with-ba4d693a app.kubernetes.io/part-of:llminferenceservice kserve.io/component:workload llm-d.ai/role:both pod-template-hash:766cc944c5]}"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'tokenizer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 INFO 06-15 07:00:07 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 06-15 07:00:07 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] 2026-06-15 07:00:09,019 [INFO] [root] TokenizationServiceServicer initialized [e2e-llm-inference-service] 2026-06-15 07:00:09,019 [INFO] [root] gRPC reflection disabled (set `ENABLE_GRPC_REFLECTION=1` to enable) [e2e-llm-inference-service] 2026-06-15 07:00:09,019 [INFO] [root] gRPC server configured to listen on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 07:00:09,020 [INFO] [root] gRPC server started on /tmp/tokenizer/tokenizer-uds.socket [e2e-llm-inference-service] 2026-06-15 07:00:09,020 [INFO] [root] Probe server started on port 8082 [e2e-llm-inference-service] 2026-06-15 07:00:09,020 [INFO] [root] Server started. [e2e-llm-inference-service] 2026-06-15 07:00:10,080 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:10,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:40,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:40 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:45,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:00:50,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:00:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:30,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:40,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:45 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:01:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:01:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:00 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:02:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:02:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:10,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:10 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:15 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:30,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:40 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:45 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:03:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:03:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:30,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:04:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:04:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:00,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:10,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:15,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:20 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:30,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:40,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:45,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:05:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:05:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:20,427 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:40,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:45 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:06:50,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:06:50 +0000] "GET /healthz HTTP/1.1" 200 261 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:00,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:00 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:10,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:10 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:15,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:30,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:40 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:07:50,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:07:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:00,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:00 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:20,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:40,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:45,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:08:50,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:08:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:15,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:40 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:09:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:09:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:00 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:20,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:40 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:10:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:10:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:10 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:15,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:30,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:30,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:40,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:45 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:11:50,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:11:50 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:00 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:20,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:12:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:12:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:00 +0000] "GET /healthz HTTP/1.1" 200 262 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:15,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:20,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:20 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:45,077 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:13:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:13:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:00,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:10,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:20,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:40,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:14:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:14:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:00,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:10,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:20,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:30,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:15:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:15:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:00,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:00,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:00 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:10,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:10 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:15,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:15 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:20,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:20 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:30,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:30 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:30,425 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:30 +0000] "GET /healthz HTTP/1.1" 200 263 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:40,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:40 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:45,078 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:45 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] 2026-06-15 07:16:50,426 [INFO] [aiohttp.access] 10.133.0.2 [15/Jun/2026:07:16:50 +0000] "GET /healthz HTTP/1.1" 200 264 "-" "kube-probe/1.34" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 24297c05-50dc-45b2-9c67-5202c79208ac [e2e-llm-inference-service] resourceVersion: '66955' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.46 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz [e2e-llm-inference-service] uid: d2addb78-1f4e-4983-8983-218180355838 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c5594c05-a8b5-4535-bf60-e2e2fea7e9f2 [e2e-llm-inference-service] resourceVersion: '67994' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.50 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 [e2e-llm-inference-service] uid: 8572c5e6-3714-4917-b782-31ec52c11134 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8572c5e6-3714-4917-b782-31ec52c11134 [e2e-llm-inference-service] resourceVersion: '67992' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766cc944c5 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.50/23"],"mac_address":"0a:58:0a:86:00:32","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.50/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.50\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:32\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5 [e2e-llm-inference-service] uid: 32f9748e-052a-43d4-bd64-34af57930dcb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-226 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"32f9748e-052a-43d4-bd64-34af57930dcb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.50"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-9mk6p [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] - name: kube-api-access-9mk6p [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n ucx_hcas=()\n for hca in \"${active_hcas[@]}\"\ [e2e-llm-inference-service] ; do\n ucx_hcas+=(\"${hca}:1\")\n done\n\n # Check if we found any active\ [e2e-llm-inference-service] \ HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements\ [e2e-llm-inference-service] \ with a comma\n hcas=$(IFS=,; echo \"${active_hcas[*]}\")\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Setting active HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n\ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if counts are equal - prefer lower index number\n\ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8000\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: kube-api-access-9mk6p [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:00:24Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] hostIP: 10.0.128.226 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.226 [e2e-llm-inference-service] podIP: 10.134.0.50 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.50 [e2e-llm-inference-service] startTime: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T07:00:24Z' [e2e-llm-inference-service] containerID: cri-o://e4b50a533016f1dcb35821769f2f190ad25dc25c902839f4a374e6efec3cd120 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://e4b50a533016f1dcb35821769f2f190ad25dc25c902839f4a374e6efec3cd120 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-9mk6p [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T07:00:24Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0 [e2e-llm-inference-service] containerID: cri-o://7e2a1e01bf22fe3caafc6483f1651317c4ec17ef08f1dcde0005e59880e149d0 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-9mk6p [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-86f69d9999- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d2addb78-1f4e-4983-8983-218180355838 [e2e-llm-inference-service] resourceVersion: '66953' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 86f69d9999 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.46/23"],"mac_address":"0a:58:0a:85:00:2e","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.46/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.46\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:2e\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-86f69d9999 [e2e-llm-inference-service] uid: c7ec8574-3741-403b-a777-99db38917f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-25 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c7ec8574-3741-403b-a777-99db38917f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.46"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-5kcg6 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-5kcg6 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-5kcg6 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] - name: kube-api-access-5kcg6 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-9nvdk [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:00:01Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] hostIP: 10.0.141.25 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.25 [e2e-llm-inference-service] podIP: 10.133.0.46 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.46 [e2e-llm-inference-service] startTime: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] finishedAt: '2026-06-15T07:00:01Z' [e2e-llm-inference-service] containerID: cri-o://b08e52af2d09e251985984fb269bc1130bce1d7700bf5a1263c10e91787b198b [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:002b0d8b8a0a27ede61dd8a8fe85971fe09fa0abcbb90ad99f092e41c4fb46a7 [e2e-llm-inference-service] containerID: cri-o://b08e52af2d09e251985984fb269bc1130bce1d7700bf5a1263c10e91787b198b [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-5kcg6 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T07:00:01Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-scheduler@sha256:88de279c6eb6758a4c600de9730e49e46b04c392846afedd03d82447379c9e7a [e2e-llm-inference-service] containerID: cri-o://38a75b8b7c9c2a6bee7e5522f89ae945d6a240936f3b0b2b060859d029ba1613 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-5kcg6 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-06-15T07:00:01Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-uds-tokenizer@sha256:aed091a51f3d64458f1fdb451d21f745186bb4517a7ba0c49913a0c617366a3e [e2e-llm-inference-service] containerID: cri-o://cb308918fa53aef9fd362cfb693f2ceb35813fe1459d600b38914cb5df0685c4 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-5kcg6 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8a3839bd-48aa-4926-98f1-288f458e8f19 [e2e-llm-inference-service] resourceVersion: '66389' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-9nvdk [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-9nvdk"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjfwp"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-9nvdk [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjfwp [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-9nvdk [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cbda1a17-45f8-4405-a56f-da2c9022375a [e2e-llm-inference-service] resourceVersion: '66408' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.88.87 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.88.87 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c25055fa-776b-41cc-88d2-336d8b97d4b0 [e2e-llm-inference-service] resourceVersion: '66382' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.20.138 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.20.138 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: acadaa95-e45b-44bb-905a-66dc21d09de1 [e2e-llm-inference-service] resourceVersion: '67999' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6d628253-aec6-4d77-ba28-2a4ea7af3a3c [e2e-llm-inference-service] resourceVersion: '66957' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] lastTransitionTime: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-86f69d9999" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 32f9748e-052a-43d4-bd64-34af57930dcb [e2e-llm-inference-service] resourceVersion: '67997' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766cc944c5 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] uid: acadaa95-e45b-44bb-905a-66dc21d09de1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"acadaa95-e45b-44bb-905a-66dc21d09de1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766cc944c5 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766cc944c5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n ucx_hcas=()\n\ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n ucx_hcas+=(\"${hca}:1\")\n\ [e2e-llm-inference-service] \ done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]}\ [e2e-llm-inference-service] \ -gt 0 ]; then\n # Join the array elements with a comma\n hcas=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n echo \"[Infer RoCE] Setting active\ [e2e-llm-inference-service] \ HCAs: ${hcas}\"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${hcas}}\n \ [e2e-llm-inference-service] \ export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${ucx_hcas}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${ucx_hcas}}\n\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if counts are equal - prefer lower index number\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using hardcoded GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-86f69d9999 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c7ec8574-3741-403b-a777-99db38917f8f [e2e-llm-inference-service] resourceVersion: '66956' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 86f69d9999 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] uid: 6d628253-aec6-4d77-ba28-2a4ea7af3a3c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6d628253-aec6-4d77-ba28-2a4ea7af3a3c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"TOKENIZERS_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8082,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/.cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models/base"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:workingDir: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 86f69d9999 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 86f69d9999 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.7.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:ba8edcbfb3f9312d158be16483785d7654e60c7090f262c42214fd2b29effada [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.1 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-uds-tokenizer:v0.7.1 [e2e-llm-inference-service] workingDir: /mnt/models [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: health [e2e-llm-inference-service] containerPort: 8082 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKENIZERS_DIR [e2e-llm-inference-service] value: /mnt/models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] mountPath: /.cache [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models/base [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 15 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /healthz [e2e-llm-inference-service] port: 8082 [e2e-llm-inference-service] scheme: HTTP [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 30 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1ece1de3-7290-45e4-87b0-1bf44651b0d8 [e2e-llm-inference-service] resourceVersion: '66399' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 93546617-a110-41b5-aa22-b39ae1b52b9e [e2e-llm-inference-service] resourceVersion: '66396' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service-jkhxd [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3f15fc38-8123-44e8-a97a-61e8e84c650c [e2e-llm-inference-service] resourceVersion: '66954' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] uid: cbda1a17-45f8-4405-a56f-da2c9022375a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T07:00:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cbda1a17-45f8-4405-a56f-da2c9022375a"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.46 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz [e2e-llm-inference-service] uid: d2addb78-1f4e-4983-8983-218180355838 [e2e-llm-inference-service] nodeName: ip-10-0-141-25.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-sv7cch7 [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2826b585-fda5-4c71-8353-d4d1d3565fbf [e2e-llm-inference-service] resourceVersion: '67995' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] uid: c25055fa-776b-41cc-88d2-336d8b97d4b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T07:01:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c25055fa-776b-41cc-88d2-336d8b97d4b0"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.50 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 [e2e-llm-inference-service] uid: 8572c5e6-3714-4917-b782-31ec52c11134 [e2e-llm-inference-service] nodeName: ip-10-0-128-226.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1ece1de3-7290-45e4-87b0-1bf44651b0d8 [e2e-llm-inference-service] resourceVersion: '66399' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 93546617-a110-41b5-aa22-b39ae1b52b9e [e2e-llm-inference-service] resourceVersion: '66396' [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T07:00:21Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66815' [e2e-llm-inference-service] uid: 8b69e049-ebb0-424c-a22e-628e9e142e21 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:01Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T07:00:21Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66815' [e2e-llm-inference-service] uid: 8b69e049-ebb0-424c-a22e-628e9e142e21 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:01Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66790' [e2e-llm-inference-service] uid: a6318305-a701-4fc5-9337-ab2bfbd79fa7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:00:02Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:02Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-06-15T07:00:06Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66598' [e2e-llm-inference-service] uid: 279cf994-ee51-4c18-9dbb-7a2f777cbd32 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:04Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-06-15T07:00:06Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66435' [e2e-llm-inference-service] uid: 49c6a7f3-efd7-48e9-88f0-92e4469fdad7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66797' [e2e-llm-inference-service] uid: 05b3a2ce-c20f-4bd8-9de3-319c637d1a26 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66459' [e2e-llm-inference-service] uid: f34e6012-e6ea-41f7-8aba-412759f662b1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66435' [e2e-llm-inference-service] uid: 49c6a7f3-efd7-48e9-88f0-92e4469fdad7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66797' [e2e-llm-inference-service] uid: 05b3a2ce-c20f-4bd8-9de3-319c637d1a26 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66459' [e2e-llm-inference-service] uid: f34e6012-e6ea-41f7-8aba-412759f662b1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66435' [e2e-llm-inference-service] uid: 49c6a7f3-efd7-48e9-88f0-92e4469fdad7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:20Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66797' [e2e-llm-inference-service] uid: 05b3a2ce-c20f-4bd8-9de3-319c637d1a26 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T07:00:00Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66459' [e2e-llm-inference-service] uid: f34e6012-e6ea-41f7-8aba-412759f662b1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"deff36ca-1b3a-4c5c-9466-df79107d57e8"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-06-15T06:59:59Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: deff36ca-1b3a-4c5c-9466-df79107d57e8 [e2e-llm-inference-service] resourceVersion: '66417' [e2e-llm-inference-service] uid: ea9085c4-6d1b-410f-832e-dfabe887909b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:16:53Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 766cc944c5 [e2e-llm-inference-service] timestamp: '2026-06-15T07:16:49Z' [e2e-llm-inference-service] window: 17.139s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 108047727n [e2e-llm-inference-service] memory: 2319176Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1165 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1166 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-06-15T07:16:53Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 86f69d9999 [e2e-llm-inference-service] timestamp: '2026-06-15T07:16:38Z' [e2e-llm-inference-service] window: 19.54s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 58540923n [e2e-llm-inference-service] memory: 29476Ki [e2e-llm-inference-service] - name: tokenizer [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 220777n [e2e-llm-inference-service] memory: 362320Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-06-15T07:16:53.314085] end - ❌ 1030.788s: Service returned 401: [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:244: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 7 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 7 failed, 28 passed, 3 skipped, 23 warnings in 5514.93s (1:31:54) ======= [must-gather] [must-gather ] OUT 2026-06-15T07:33:37.779260875Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 52b23002-b21e-42e1-a029-20bb4a09421e [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.20" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-06-15T07:33:37.830865603Z namespace/openshift-must-gather-5jmhk created [must-gather] [must-gather ] OUT 2026-06-15T07:33:37.83819193Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-52c67 created [must-gather] [must-gather ] OUT 2026-06-15T07:33:37.865001879Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:44.543182231Z [disk usage checker] Started [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:44.546728183Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:44.818130227Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:44.991012510Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:44.991194758Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:44.991194758Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:44.993723012Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.160141787Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.160171191Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.160171191Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.162700004Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.327834340Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.327930789Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.327930789Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.330611535Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.495555515Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.495585259Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.495585259Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.498008184Z Error getting logs from rhods-notebooks [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.663921062Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.663950951Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.663950951Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.666641888Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.831428547Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.831491477Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.831491477Z namespaces "istio-system" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:45.834046937Z Error getting logs from istio-system [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.000108232Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.000136519Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.000136519Z namespaces "knative-serving" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.002591204Z Error getting logs from knative-serving [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.164186303Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.164305253Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.164305253Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.166725673Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.452042047Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.729190157Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.877529184Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.959929768Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:46.962264703Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.100170686Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.186562827Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.188953761Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.337433168Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.418466585Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.420653624Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.556827343Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.636774228Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.638867673Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.773333506Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.857621862Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.859968083Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:47.994613671Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.076995132Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.079494044Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.218619973Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.301202383Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.303850710Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.443185047Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.532142040Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.534673039Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.672540575Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.755572919Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.757961569Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.894476338Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.981991602Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:48.984402680Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.124749397Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.208452873Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.210803595Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.352485066Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.435895887Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.438591084Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.552228316Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.577096599Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.663356633Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.665873826Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.800303171Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.882486354Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:49.884767178Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.023526750Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.105383200Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.107743506Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.243997691Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.328887769Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.331318983Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.467395145Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.552531880Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.554875257Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.688398362Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.776301597Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:50.779024384Z Error collecting info from [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:51.499796617Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:51.634033314Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:51.771021005Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:51.908067444Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:52.044644321Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:52.667954005Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:52.802721436Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:52.946428132Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:53.084963239Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:53.221266038Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:53.798167886Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:54.557323287Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:54.967498758Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:55.708410471Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027734133Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027782437Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027782437Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027782437Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027782437Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027782437Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027782437Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027782437Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.027782437Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.032511671Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:56.191188569Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:57.148605410Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:57.897421621Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231319936Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231361571Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231361571Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231361571Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231361571Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231361571Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231361571Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231361571Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.231361571Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.236546341Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:58.400508600Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:59.496224028Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:33:59.562016945Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.217866294Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551147688Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551194070Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551194070Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551194070Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551194070Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551194070Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551194070Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551194070Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.551194070Z [previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found, container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.557105354Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:00.721578178Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:01.662016218Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.388655354Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734428815Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734473782Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734473782Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734473782Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734473782Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734473782Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734473782Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734473782Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.734473782Z [previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found, container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.740772985Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:02.903062730Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:03.894152494Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:04.572446654Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:04.632846289Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.148906426Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.149003797Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.149003797Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.149003797Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.149003797Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.149003797Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.149003797Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.149003797Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.149003797Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.154627763Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:05.325312636Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:06.298181945Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.106719496Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484619862Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484663246Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484663246Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484663246Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484663246Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484663246Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484663246Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484663246Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.484663246Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.490666204Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:07.667378961Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:08.709681795Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.451132218Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.577353315Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788394991Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788440610Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788440610Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788440610Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788440610Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788440610Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788440610Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788440610Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.788440610Z [previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found, container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.794277975Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:09.962506703Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:10.913373782Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.648446251Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993386932Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993438232Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993438232Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993438232Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993438232Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993438232Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993438232Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993438232Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.993438232Z [previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found, container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:11.998878548Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:12.164311241Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:13.132563774Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:13.898196705Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242468252Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242513877Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242513877Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242513877Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242513877Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242513877Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242513877Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242513877Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.242513877Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.248493162Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.409008799Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:14.582176719Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:15.437887659Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.170765653Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492463488Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492510466Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492510466Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492510466Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492510466Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492510466Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492510466Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492510466Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.492510466Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.497720201Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:16.663610702Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:18.010714430Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:18.743783317Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059057199Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059115975Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059115975Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059115975Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059115975Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059115975Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059115975Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059115975Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.059115975Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.064711325Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.236295172Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:19.587057629Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:20.233178201Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.006928732Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344050878Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344118876Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344118876Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344118876Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344118876Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344118876Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344118876Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344118876Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.344118876Z [previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found, container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.348655701Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:21.508536501Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:22.473897074Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.214286335Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547120400Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547167307Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547167307Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547167307Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547167307Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547167307Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547167307Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547167307Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.547167307Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.552518892Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:23.714025085Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:24.593719421Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:24.788388654Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.519336381Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864502230Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864549754Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864549754Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864549754Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864549754Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864549754Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864549754Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864549754Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.864549754Z [previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found, container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:25.869927234Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:26.035003913Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:27.273267531Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:27.435926890Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:27.809961636Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:27.978912686Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:29.041863431Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:29.598586103Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:29.777713578Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140698282Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140756088Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140756088Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140756088Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140756088Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140756088Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140756088Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140756088Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.140756088Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.145735569Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:30.311281423Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:31.315390403Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.059617321Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379459421Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379505006Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379505006Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379505006Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379505006Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379505006Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379505006Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379505006Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.379505006Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.384919981Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:32.546053741Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:33.480334922Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.205172913Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518613162Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518664797Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518664797Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518664797Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518664797Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518664797Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518664797Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518664797Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.518664797Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.523743591Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.603554571Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:34.683790263Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:35.624536398Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.337986422Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676162057Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676210463Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676210463Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676210463Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676210463Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676210463Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676210463Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676210463Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.676210463Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.681655436Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:36.847747951Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:37.804159965Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.524661523Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845399878Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845441616Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845441616Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845441616Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845441616Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845441616Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845441616Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845441616Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.845441616Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:38.850961453Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:39.012351675Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:39.608328823Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:39.942847706Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:40.667241888Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009286518Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009330389Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009330389Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009330389Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009330389Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009330389Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009330389Z [container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing, previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009330389Z [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.009330389Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.014565358Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:41.173800582Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:42.361144734Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:42.537151897Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:42.857479524Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:43.024460036Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:44.166181964Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:44.321258766Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:44.613073158Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:44.631108943Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:44.792279244Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:46.040323691Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:46.197240983Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:46.531835066Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:46.697120950Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:47.237602798Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:47.426408054Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:47.754555275Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:47.922645551Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:48.481598136Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:48.731899801Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:49.073987647Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:49.240331146Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:49.620866935Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:49.775236634Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:49.958461306Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:50.296034913Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:50.459470502Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:50.971621082Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:51.156510408Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:51.487710286Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:51.658524836Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:52.201468277Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:52.394036165Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:52.735704104Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:52.901855361Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:53.470243065Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:53.660035042Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:54.005142782Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:54.171215688Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:54.625962824Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:54.703191894Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:54.892200820Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:55.239458150Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:55.403623391Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:56.009366584Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:56.220270371Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:56.561977788Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:56.728453890Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:57.275953638Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:57.472483089Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:57.817631255Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:57.989307597Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:58.509987301Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:58.737557219Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:59.092759919Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:59.258878985Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:59.630651779Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:59.789263461Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:34:59.980200048Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:00.308191759Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:00.466995679Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:00.991976724Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:01.178260800Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:01.511392102Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:01.679733767Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:02.199864134Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:02.389229779Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:02.716056419Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:02.875960327Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:03.422193760Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:03.627742767Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:03.990044210Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:04.158515014Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:04.635677266Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:04.750540798Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:04.939057420Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:05.272247285Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:05.430301113Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:05.950118623Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:06.132925677Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:06.469348299Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:06.629614412Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:07.131808948Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:07.320165403Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:07.658076958Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:07.821252559Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:08.349194704Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:08.548838135Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:08.907345107Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:09.071280672Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:09.585632935Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:09.640627756Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:09.768523241Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:10.108036044Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:10.275964904Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:10.800649915Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:10.984836552Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:11.320680522Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:11.480400326Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:12.000199151Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:12.181114581Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:12.519832712Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:12.678187192Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:13.200534464Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:13.391734198Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:13.730193420Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:13.898772113Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:14.497964762Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:14.647217376Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:14.684213553Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:15.018023325Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:15.182398447Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:15.733981291Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:15.916897735Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:16.249369594Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:16.417999972Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:16.942457446Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:17.131199439Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:17.476388384Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:17.640344041Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:18.165818703Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:18.356011215Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:18.686702324Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:18.851558661Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:19.387868853Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:19.586388768Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:19.652778509Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:19.921028104Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:20.091277688Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:20.668819800Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:20.858120922Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:21.186495299Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:21.351609939Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:21.872617063Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:22.053318738Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:22.385176170Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:22.554204783Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:23.070749145Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:23.257548000Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:23.585680419Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:23.772320828Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:24.307257454Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:24.490019617Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:24.659229059Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:24.834231658Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:25.002400472Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:25.520471444Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:25.709035600Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:26.048554632Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:26.211007621Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:26.735460201Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:26.924467747Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:27.256525366Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:27.418693623Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:27.964330586Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:28.145718132Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:28.476511508Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:28.648602346Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:29.174194995Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:29.416554824Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:29.664137860Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:29.754664821Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:29.915047619Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:30.464137022Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:30.663005528Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:31.006060468Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:31.172350337Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:31.761346242Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:31.956260192Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:32.315207672Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:32.481803583Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:33.011963544Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:33.211096821Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:33.589315714Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:33.754109969Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:34.269735585Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:34.457433591Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:34.668933084Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:34.799678366Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:34.972572237Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:35.520436383Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:35.710287260Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:36.051324462Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:36.228136332Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:36.758112671Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:36.947204613Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:37.285989664Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:37.454644661Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:38.000358168Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:38.196551894Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:38.538000827Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:38.701370598Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:39.292662934Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:39.492281639Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:39.673785556Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:39.825183598Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:39.994241493Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:40.585010291Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:40.779615532Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:41.119761470Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:41.285304060Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:41.875014362Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:42.057357439Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:42.402923832Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:42.565828832Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:43.091093490Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:43.271316425Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:43.617838595Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:43.778896285Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:44.308605027Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:44.501327162Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:44.678521671Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:44.840341617Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:45.007376120Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:45.526027395Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:45.710556619Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:46.063003642Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:46.230609233Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:46.784056214Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:46.968523413Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:47.308107193Z Wrote inspect data to must-gather. [must-gather] [must-gather-xnhfm] POD 2026-06-15T07:35:47.329237388Z Caches written to disk [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.337078689Z waiting for gather to complete [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.345213948Z downloading gather output [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.649869516Z receiving incremental file list [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.668663907Z ./ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.668734999Z aggregated-discovery-api.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.669019876Z aggregated-discovery-apis.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.669781375Z event-filter.html [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.671664321Z timestamp [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.671848465Z version [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.680972999Z cluster-scoped-resources/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.680986299Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.68099209Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681078472Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681262566Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681272306Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681343538Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681498982Z namespaces/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681508632Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681541583Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681617815Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681630265Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681715077Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.681730757Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.68184123Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.682826524Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.683726606Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.683763057Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.683809288Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.683840479Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.6838818Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.683960982Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.683998953Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.684038434Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.684119666Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.684169597Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.684203698Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.684320661Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.684493775Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.688081143Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.688139575Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.68916067Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.689265952Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.689619851Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.689799935Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.689859607Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690149224Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690159184Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690163574Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690174275Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690187855Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690281887Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690359439Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69038591Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.6903933Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69041631Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690433121Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690443331Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/auth-enabled-test-kserve-85d86d876c-vrqhw.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690580614Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690591755Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690596485Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690604955Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/current.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690719938Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69078613Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690854971Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690883042Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690888062Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690891782Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.690926923Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691029505Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691091157Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691120888Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691163279Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691312552Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691320152Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691323643Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691358404Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691786074Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691856066Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691890897Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691898177Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691903537Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.691931627Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69202719Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692096502Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692126392Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692130923Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692141283Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692176144Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692590954Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692657535Z namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692713457Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692771828Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692910022Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692920662Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692924662Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.692945612Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/current.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693015084Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693089576Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693151057Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693187948Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693196258Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693199869Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69324485Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693316652Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693386613Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693424764Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693449395Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693591308Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693598648Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693607949Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693637709Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693779413Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693865245Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693879585Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693883785Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693887555Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.693916626Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694109891Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694240494Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694258235Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694262425Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694266345Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694269855Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694442539Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69450303Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694534551Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694582903Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694771497Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694784797Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694791498Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.694813118Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695259069Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695331941Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695351841Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695358702Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695365352Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695397383Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695490465Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695551206Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695589167Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695620538Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695795552Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695811553Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695816473Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695833203Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.695952036Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696055939Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696091789Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.6960998Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69610341Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696131981Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696230103Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696291124Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696326775Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696333295Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696337066Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696363566Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696591222Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696658733Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696712745Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696770136Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696851988Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696861128Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696865748Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.696896679Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697004642Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697070133Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697109445Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697193807Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697273578Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697283849Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697288519Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69732015Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697419942Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697479254Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697502694Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697558596Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697696609Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697707759Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697711549Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69774926Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.697918874Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698007047Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698011647Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698015237Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698018747Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698023757Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698381876Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698446087Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698482638Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698538329Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698666673Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698700753Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698716084Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698729254Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698862597Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.698931619Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69895621Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69896806Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.69897228Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699015031Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699461882Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699527034Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699543284Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699593935Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/router-gateway-1-openshift-default-75dcfd69c9-dh6qf.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699735219Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699748709Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/istio-proxy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699753459Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699761139Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.699982145Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700067127Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700092278Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/router-gateway-2-openshift-default-78c98f6f4c-ddrqp.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70019043Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70019948Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/istio-proxy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70020365Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700232261Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700335504Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700414406Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700467967Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/router-with-refs-pd-test-kserve-6f78896447-wshh4.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700639591Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700650521Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700660092Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.700688082Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701060431Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701118553Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701158764Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701169294Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701173284Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701194025Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701511122Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701570474Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701615635Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701623445Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701628125Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701640996Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70183084Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701849251Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701892192Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.701927232Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702045855Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702054336Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702058126Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702069586Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702456405Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702515827Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702552068Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702560278Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702564368Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702607409Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702734482Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702804254Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702818454Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.702872406Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703016839Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70302715Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70303197Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70303944Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703186943Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703252815Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703275246Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703281656Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703287226Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703313866Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703411029Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70346983Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703508201Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703524142Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703528112Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703553152Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703833719Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703902221Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703933382Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.703973803Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/router-with-refs-test-kserve-578d595fc-gtvkx.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704080735Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704088165Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704092136Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704101086Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704738861Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704808363Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704847334Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704853794Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704857394Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704877245Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.704990708Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70510277Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705136601Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705169652Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705316366Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705323976Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705327876Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705358997Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70549466Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705567822Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705599802Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705606093Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705609903Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705643864Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705834408Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70591146Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705931901Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705946441Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705950351Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.705990042Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706348191Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706408022Z namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706442073Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706481724Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706519625Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706556496Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706606027Z namespaces/kuadrant-system/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706657779Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706764851Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706810592Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706845433Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706884154Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.706961066Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707266734Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70752866Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707568271Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707611772Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707650623Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707733845Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707818877Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707872588Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707903769Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.707992031Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.708017452Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.708067223Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.708254758Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70836753Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.708808731Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.708898514Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.709208711Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.709329364Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.709544679Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.709656862Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.709712694Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.709836846Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.709856937Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.709909758Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.70995782Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710031941Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710074182Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710109053Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710142874Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710177445Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710226706Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710241267Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710248527Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino-686db986cb-n5rxl.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710340089Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710349679Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.710353679Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.71038519Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.712590174Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.712651726Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.712667636Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.712750708Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/authorino-operator-6d75c86569-cxdzr.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.71284249Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.712860071Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.712864861Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.712874201Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713051985Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713100286Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713143327Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713166408Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/dns-operator-controller-manager-65b49595d7-knbkx.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.71326553Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713275641Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713280501Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713287761Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713399244Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713466315Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713501966Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713527737Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin-7dbb555447-w6b6b.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713604049Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713611509Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713615729Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.71364751Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713750972Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713814344Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713859385Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.713885256Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.714006349Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.714016379Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.714020949Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.714032059Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.723962863Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724010134Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724054215Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724088326Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador-limitador-69574b596d-qnf8x.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724188548Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724198609Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724203769Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724221359Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724336162Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724400963Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724417814Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724458315Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/limitador-operator-controller-manager-6f9f468797-cgn2h.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724572778Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724582148Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724585688Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724627869Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724791433Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724852745Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724877985Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724916986Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.724982028Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725012378Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.72504685Z namespaces/openshift-ingress/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725112511Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725192543Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725246654Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725313336Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725345377Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725432379Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725692785Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.725971422Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726223098Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726232368Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726241789Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726248279Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726253749Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726320961Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726346351Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726434724Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726450224Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726596537Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.726978167Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.7270882Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.727303085Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.727396097Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.727655663Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.727816697Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728050563Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728152346Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728201427Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728285069Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.72834208Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728400001Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728452523Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728553775Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728596536Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728656998Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.72873944Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728793661Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728832902Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728842612Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.728895024Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/istiod-openshift-gateway-75c67f8887-qbmcr.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.729003456Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.729011836Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.729016107Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.729068548Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.76096312Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.761026921Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.761051532Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.761117383Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.761234446Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.761243447Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.761247967Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.761286418Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.762730743Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.76299673Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763035511Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router-default-8bdfdcbd8-4fc26.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763143053Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763152073Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763156804Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763164524Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/current.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763304747Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/previous.insecure.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763390349Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/previous.log [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.76342074Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763481892Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763539343Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.763603544Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.770957905Z [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.770973515Z sent 5,759 bytes received 1,598,567 bytes 3,208,652.00 bytes/sec [must-gather] [must-gather-xnhfm] OUT 2026-06-15T07:35:48.770978165Z total size is 20,002,980 speedup is 12.47 [must-gather] [must-gather ] OUT 2026-06-15T07:35:48.961785893Z namespace/openshift-must-gather-5jmhk deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 52b23002-b21e-42e1-a029-20bb4a09421e [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.20" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-06-15T07:35:49.082138273Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:447854878808c34eb4f7e1488a1e3009b8bc3d2e92b4a5d467c6bdee045d7922 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 52b23002-b21e-42e1-a029-20bb4a09421e [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.20" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-06-15T07:35:49.10156496Z namespace/openshift-must-gather-d7tsl created [must-gather] [must-gather ] OUT 2026-06-15T07:35:49.112454156Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-r2998 created [must-gather] [must-gather ] OUT 2026-06-15T07:35:49.147953937Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:447854878808c34eb4f7e1488a1e3009b8bc3d2e92b4a5d467c6bdee045d7922 created [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:50.412006058Z [disk usage checker] Started [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:50.415910504Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:50.635154948Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:50.810236324Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:50.923925741Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:51.032727405Z Gathering data for ns/default... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:51.396756743Z Gathering data for ns/openshift... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:51.916670340Z Gathering data for ns/kube-system... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:52.783309064Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.170138228Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.170368947Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.170368947Z namespaces "assisted-installer" not found [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.415512865Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.423400988Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.428881137Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.431994934Z Executing Istio gather script [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.442152320Z INFO: Gathering HAProxy config files [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.444005143Z INFO: Collecting host service logs for crio [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.444390520Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.444706888Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.445008352Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.445347917Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.445650757Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.445965616Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.446283396Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.446592218Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.446899749Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.447217374Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.447580766Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.450140903Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.512777297Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.774037554Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.876046581Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:53.913291807Z No resources found [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.489471929Z INFO: Collecting Insights Archives from insights-operator-5bbd86d6bd-4bc2h [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.489514805Z insights-runtime-extractor-bx7hl [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.489534417Z insights-runtime-extractor-mrjn4 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.489553920Z insights-runtime-extractor-sfxtc [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.509045604Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.688203158Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.688300730Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.694154165Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.727339635Z Inspecting resource ns/openshift-operators [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.734353793Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:54.942425418Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.112444828Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.112444828Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.123951733Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.126343561Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.198102352Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.352845979Z Gathering data for ns/openshift-operators... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.386934572Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.440948781Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.628726619Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:55.882391541Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.146676102Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.157875171Z No resources found [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.230821863Z INFO: 'previous-logs' folder not found on ip-10-0-128-226.ec2.internal, skipping... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.301278825Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.373801201Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.467877582Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.536760111Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.646203753Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.811458524Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.889894171Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.914129530Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.950641673Z tar: Removing leading `/' from member names [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:56.954966314Z INFO: HAProxy config collection complete. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.041096037Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.138001478Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.325847143Z INFO: 'previous-logs' folder not found on ip-10-0-128-243.ec2.internal, skipping... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.355761522Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.376891448Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.393596490Z Inspecting resource clusterserviceversion in namespace openshift-operators [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.439948031Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.504348933Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.575234053Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:57.979178379Z W0615 07:35:57.979128 1132 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.009127754Z INFO: 'previous-logs' folder not found on ip-10-0-141-25.ec2.internal, skipping... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.009233014Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.009283102Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.010706554Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.019465292Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.020542621Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.176921631Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.260801816Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.418635958Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.510789462Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.537562996Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.594719666Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.647628305Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.901350061Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.908922799Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.996650725Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:58.999789300Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.023141607Z INFO: INTERCONNECT MODE [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.023223623Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.275220757Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.275636192Z INFO: Gathering OVN_Northbound from ovnkube-node-f4bzh... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.276964733Z INFO: Gathering OVN_Northbound from ovnkube-node-r4jhk... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.276964733Z INFO: Gathering OVN_Northbound from ovnkube-node-rkdkp... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.277816669Z INFO: Gathering OVN_Southbound from ovnkube-node-f4bzh... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.277816669Z INFO: Gathering OVN_Southbound from ovnkube-node-r4jhk... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.277816669Z INFO: Gathering OVN_Southbound from ovnkube-node-rkdkp... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.277816669Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.387841705Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.393553427Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.611499056Z tar: Removing leading `/' from member names [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.793957773Z tar: Removing leading `/' from member names [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.795019936Z tar: Removing leading `/' from member names [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.810266865Z tar: Removing leading `/' from member names [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.857763288Z tar: Removing leading `/' from member names [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.875178760Z tar: Removing leading `/' from member names [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.886782436Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.896672442Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.899251176Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:35:59.929151381Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.191068406Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.195447758Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.258115516Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.295816867Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.426621268Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.427739381Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.428955945Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.431369505Z 6.2M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.434295123Z ovnk_database_store/ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.434362205Z ovnk_database_store/ovnkube-node-f4bzh_sbdb [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.480937359Z ovnk_database_store/ovnkube-node-r4jhk_sbdb [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.502072037Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.505641423Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.517193968Z ovnk_database_store/ovnkube-node-rkdkp_sbdb [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.561792953Z ovnk_database_store/ovnkube-node-f4bzh_nbdb [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.578358005Z ovnk_database_store/ovnkube-node-rkdkp_nbdb [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.590487995Z ovnk_database_store/ovnkube-node-r4jhk_nbdb [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.590529092Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.605243610Z INFO: Network log collection complete. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.721402977Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.839452840Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:00.932015220Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:01.057899942Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:01.265119337Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:01.586319813Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:01.589250105Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:02.097676155Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:02.124803348Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:02.271840286Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d66a6b2ec52dede5d83ce04f6abfc519df1fd7dc36248741e03ecb9bf8d1762 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:02.399774631Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:02.563475586Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:02.714797812Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:03.173390441Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:03.241054951Z Gathering data for ns/openshift-console... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:03.367995641Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:03.604495635Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:03.608108223Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:03.720284111Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:03.883729580Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:03.993553579Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:04.144841740Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:04.147070214Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:04.337652405Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:04.588595392Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:04.591309943Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:04.714352331Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:04.821811034Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:04.882248096Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.070291600Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.073291194Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.166254697Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.290116599Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.501988912Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.504386603Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.709471056Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.726278525Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.928216213Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:05.930985235Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.044884060Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.114707855Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.171347768Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.380333475Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.383961629Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.613065502Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.811160213Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.879815106Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:06.883434523Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.069864317Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.195852883Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.304600603Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.307130581Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.392928698Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.490011724Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.732652161Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.735358694Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:07.913525879Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.120819075Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.123499111Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.331650069Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.334976995Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.541967634Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.544459270Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.745466091Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.984338284Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:08.987931254Z Inspecting resource crd/istiocnis.sailoperator.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.100709994Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.189977296Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.430232880Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.433575643Z Inspecting resource crd/istiorevisions.sailoperator.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.497220166Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.660125603Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.752368316Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.964558607Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:09.967199592Z Inspecting resource crd/istiorevisiontags.sailoperator.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.164354739Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.390186248Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.392799887Z Inspecting resource crd/istios.sailoperator.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.645680203Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.712529760Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.731561812Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.936744359Z Collecting performance related data for node ip-10-0-141-25.ec2.internal [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.946720700Z Collecting performance related data for node ip-10-0-128-226.ec2.internal [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.958111926Z Collecting performance related data for node ip-10-0-128-243.ec2.internal [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.962617492Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:10.966203733Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:11.182187102Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:11.442175558Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:11.451499500Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:11.678419691Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:11.960998108Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:11.965350857Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:12.193179243Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:12.506566094Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:12.510370509Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:12.763814310Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.030376108Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.032967160Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.237976297Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.308217203Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.470166756Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.473812619Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.689973856Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.941964793Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:13.945268559Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:14.049369852Z Gathering data for ns/kserve... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:14.174460719Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:14.413817919Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:14.416744911Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:14.669137416Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:14.912288696Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:14.915110920Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:15.118075497Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:15.356110389Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:15.358438957Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:15.574320863Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:15.736504905Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:15.799714490Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:15.802019228Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:16.013390167Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:16.243321265Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:16.245789196Z Inspecting resource crd/ztunnels.sailoperator.io [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:16.501469240Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:16.503009366Z Gathering data for ns/openshift-config... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:16.755629858Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:16.903048080Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:16.916518213Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:17.140851168Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:17.496495159Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:17.890515209Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:18.220586030Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:18.281494762Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:18.401471865Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:18.537723924Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:18.680025318Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:18.683985462Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:18.937325703Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:19.085482356Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:19.533515920Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:19.924620829Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.274377544Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.314616468Z Collecting kubelet logs for node ip-10-0-141-25.ec2.internal [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.316506366Z Collecting kubelet logs for node ip-10-0-128-226.ec2.internal [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.318396190Z Collecting kubelet logs for node ip-10-0-128-243.ec2.internal [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.576195962Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.589315936Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.743909459Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.778144800Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.838676098Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-d7tsl namespace [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:20.844215064Z INFO: Node performance data collection complete. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:21.072881685Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:21.475292480Z Inspecting openshift-gateway IstioRevision [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:21.593020477Z Inspecting resource ns/openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:21.781844467Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:21.830653676Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:22.980297316Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:23.133042921Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:23.270101098Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:23.276072433Z Inspecting resource net-attach-def,roles,rolebindings in namespace openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:23.505767227Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:23.651009553Z Collecting /debug/syncz from istiod-openshift-gateway-75c67f8887-qbmcr in namespace openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:24.085359561Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:24.447944171Z Inspecting kserve-ci-e2e-test data plane namespace [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:24.448002642Z Inspecting resource ns/kserve-ci-e2e-test [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:24.600577171Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:25.737362554Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:25.751853198Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:25.882597588Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.364131642Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.571342866Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982853400Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982896375Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982896375Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982896375Z [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982896375Z [one or more errors occurred while gathering container data for pod auth-enabled-test-kserve-85d86d876c-vrqhw: [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982896375Z [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982896375Z [previous terminated container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" not found, container "main" in pod "auth-enabled-test-kserve-85d86d876c-vrqhw" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w: [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982896375Z [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.982896375Z [container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" not found]] [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:26.987515913Z Inspecting resource net-attach-def,roles,rolebindings in namespace kserve-ci-e2e-test [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:27.240917426Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:27.486385830Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:27.640410199Z Collecting Envoy config for pods in kserve-ci-e2e-test pointing to openshift-gateway revision [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:27.810320314Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:28.448233563Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:28.818702226Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:29.343758014Z Collecting config_dump and stats for pod router-gateway-1-openshift-default-75dcfd69c9-dh6qf.kserve-ci-e2e-test [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:29.452723867Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:29.846856126Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:30.336912110Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:30.423216157Z Collecting config_dump and stats for pod router-gateway-2-openshift-default-78c98f6f4c-ddrqp.kserve-ci-e2e-test [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:30.690306634Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:30.761268326Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:31.095961320Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:32.137781355Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:32.263378365Z Inspecting openshift-ingress data plane namespace [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:32.263423633Z Inspecting resource ns/openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:32.456001629Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:32.523705184Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:32.953927388Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:33.531149296Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:33.799863799Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:33.983225276Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:34.375449542Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:34.380290532Z Inspecting resource net-attach-def,roles,rolebindings in namespace openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:34.789891302Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:34.925232671Z Collecting Envoy config for pods in openshift-ingress pointing to openshift-gateway revision [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:34.934470668Z Wrote inspect data to must-gather. [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:34.934533236Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:34.934533236Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:35.285875618Z Collecting config_dump and stats for pod openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd.openshift-ingress [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:35.767861882Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.456374218Z Done executing Istio gather script [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.585925640Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.776420051Z Caches written to disk [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.456374218Z Done executing Istio gather script [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.585925640Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.776420051Z Caches written to disk [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.456374218Z Done executing Istio gather script [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.585925640Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-hlkzt] POD 2026-06-15T07:36:36.776420051Z Caches written to disk [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:37.815051714Z waiting for gather to complete [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:37.818949941Z downloading gather output [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.107580852Z receiving incremental file list [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.130324748Z ./ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.13042908Z aggregated-discovery-api.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.130911302Z aggregated-discovery-apis.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.13161656Z event-filter.html [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.1356486Z timestamp [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.135775813Z version [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138122441Z cluster-scoped-resources/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138139032Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138143272Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138181093Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138282275Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138376808Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138443759Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138506211Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138617574Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138711586Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.13886683Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.138916691Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.139034674Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.139112976Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.139431554Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.139473425Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.13965912Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.139927216Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.140139921Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.140289915Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.143524736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.145112895Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.145263939Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.145399312Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.145525355Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.14569377Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.146197492Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.146345206Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.146560121Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.146692764Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.147522935Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.147791451Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.147916375Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.148084079Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.149225227Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.149443892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.149625317Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.149808722Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.150454598Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.150646793Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.15097167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.151291238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.152071708Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.152194791Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.152304804Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.152424297Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.152533609Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.152638422Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.152827787Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.153025401Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.153860132Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.153999286Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.154132579Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.156297943Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.156467327Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.156615621Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.156890748Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.157198735Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.157696728Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.157833991Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.157960574Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.158062317Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.158167219Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.158282102Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.158391195Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.158549309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.158699402Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.158831366Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.159114683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.159261767Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.159363969Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.159918563Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.160033196Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.160151899Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.160342854Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.161078832Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.161177914Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.161361619Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.161470931Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.161629135Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.161766989Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.161919013Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.162048626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.162153738Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.162266981Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.162407065Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.162535768Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.162639931Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.162822755Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.162949258Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.163067971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.163210845Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.163567103Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.163754708Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.16987326Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.169989493Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.170823694Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.170939807Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.171054849Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.171201923Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.1714632Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.171784078Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.171911971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.172019864Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.172180328Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.17229736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.172421903Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.172557107Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.172707921Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.172840114Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.174985517Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.175422958Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.176007753Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.176186947Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.17632994Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.176480604Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.176612458Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.176811052Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.177598072Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocnis.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.177995082Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.178288539Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiorevisions.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.180103294Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiorevisiontags.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.180206307Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istios.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.181621872Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.182971585Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.183079608Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.183229302Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.183343515Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.183477158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.18357048Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.183720564Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.187042057Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.187317584Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.196925022Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.200889021Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.201044515Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.201316481Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.201474165Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.201616309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.201748752Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.201900056Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.202130862Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.202235804Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.202338927Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.202519311Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.202642605Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.202802828Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.202914891Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.203022754Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.203153987Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.203315271Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.203410084Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.203525606Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.20366049Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.203820964Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.203916856Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.204048149Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.204161942Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.204282315Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.204534691Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.204662495Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.204913811Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.205008333Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.205129566Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.205225429Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.207133726Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.207258489Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.207363042Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.207466324Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.207566357Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.207733981Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.207851414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.208000007Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.20811267Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.209535356Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.20970021Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.209808722Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.209953916Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.21012878Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.210308065Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.210538561Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.211064514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.211352821Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.211421263Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.211530145Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.211632978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.212234553Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.212423977Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.21253476Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.213926905Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.214065488Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.214211522Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.214308834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.214457338Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.214751215Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.214875248Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.215028792Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.215405051Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.215497894Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.215608567Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.215811921Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.215955705Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.216088699Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.216225122Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.216391966Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ztunnels.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.216992241Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.217162105Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.217208496Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.217307949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.217386261Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.217462963Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.217553885Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.217729159Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.217855252Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218008776Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218093808Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.21817388Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218264942Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218346824Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218437217Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218527949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218612151Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218768635Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.218859867Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.21894501Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219029521Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219111914Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219196796Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219279298Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.21936385Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219454492Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219530364Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219612936Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219721959Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219827801Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219913994Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.219998145Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220081978Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22017387Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220257242Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220337714Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220419106Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220503368Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22058457Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220667362Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220775045Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220859037Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.220942779Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221026521Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221109873Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221198245Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221282797Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22136599Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221454332Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221566935Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221661927Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22180211Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221865342Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.221958094Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222041246Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222128929Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222216001Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222300943Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222383985Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222475887Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222566609Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222651431Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222778945Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222867857Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.222967319Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223044981Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223132763Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223244816Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223329838Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223417481Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223517983Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223601065Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223708828Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223838771Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.223929133Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224017705Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224105808Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.2241926Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224281042Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224369894Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224452576Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224538298Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22462774Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224747083Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224836086Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.224926248Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.2250116Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225097122Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225185074Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225273567Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225358789Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225445481Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225538923Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225629656Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225763529Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225858321Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.225939363Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226025655Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226114717Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22620278Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226288722Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226379304Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226463776Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226549598Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226642071Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226764714Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta2.authorino.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226859646Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.226942298Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22702444Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227081012Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227091872Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227139613Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-9d4v7.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227247376Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-kwptb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227331738Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-t984z.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227474971Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227510502Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227715417Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227841511Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.227927483Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.228020725Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.228336193Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.228434185Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.228512687Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22861763Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.228766934Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.228840535Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.228941388Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22901991Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.229112442Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.229258016Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.229363018Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22944175Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.229536382Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.229632425Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.229758598Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.22984483Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.229999924Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.230084466Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.230184629Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23022856Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.230343522Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.230437815Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.230525147Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23063813Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.230776153Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.230870756Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231012059Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231099671Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231185423Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231275336Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231360538Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231489801Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231580673Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231687366Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231782438Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231883261Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.231965683Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232087326Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232172328Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.2322631Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232322062Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232352823Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232419184Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232462055Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232519377Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232566218Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232628759Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232702821Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232768933Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232813154Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232871505Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232900856Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232956027Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.232965658Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233010559Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233101981Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233181173Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233232054Z cluster-scoped-resources/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233239154Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233276766Z cluster-scoped-resources/core/nodes/ip-10-0-128-226.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233428029Z cluster-scoped-resources/core/nodes/ip-10-0-128-243.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233552682Z cluster-scoped-resources/core/nodes/ip-10-0-141-25.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233660915Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233724277Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233772928Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23386203Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.233947242Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234022544Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234106676Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234184618Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234324841Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234413494Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234500606Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234585438Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234690461Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234800763Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234882325Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.234968668Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23505609Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235146732Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235236164Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235318266Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235407219Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235493161Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235573643Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235657445Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235723926Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235811618Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235887091Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.235981643Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236059945Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236150137Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23629319Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236414464Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236499006Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236550197Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236559597Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236592638Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236707071Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236725721Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236749552Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236832254Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236867825Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236932256Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236939517Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.236977907Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23706667Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237208003Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237268525Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237275975Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237321346Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237422318Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237431789Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23746233Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237545352Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237562132Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237700705Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237739906Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237798208Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237833329Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23789654Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.237915411Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238031774Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238065514Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238113546Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238145556Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238197898Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238229649Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23828186Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238321691Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238384432Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238423753Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238493955Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238511036Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238586547Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238610908Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238655569Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23870602Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238789543Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238806583Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238921246Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238932246Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.238947066Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239028778Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23907804Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239174152Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239258244Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239409528Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.23950992Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239618193Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239747366Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239858429Z cluster-scoped-resources/operators.coreos.com/operators/servicemeshoperator3.openshift-operators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239950421Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.239994542Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240037634Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240120925Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240198678Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240335431Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240469594Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240565927Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240650849Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240843974Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.240929076Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241012548Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.24109864Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241181112Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241260084Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241341986Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241468329Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241558541Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241645503Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241757596Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.241840428Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.2419237Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242004232Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242113235Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242155886Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242253218Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242342721Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242453254Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242592187Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242693369Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242814183Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242896865Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.242982707Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243069219Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243153051Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243240643Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243324715Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243409607Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243495869Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243582722Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243689404Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243782956Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243868699Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.243963501Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244046143Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244147925Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244231208Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244309909Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244406062Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244488864Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244581436Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244660378Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244783541Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244876904Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244935895Z cluster-scoped-resources/sailoperator.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244943525Z cluster-scoped-resources/sailoperator.io/istios/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.244992526Z cluster-scoped-resources/sailoperator.io/istios/openshift-gateway.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245061418Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245071508Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.24512488Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245174291Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245183051Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245233303Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245285594Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245294384Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245334435Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245396147Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245434788Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-128-226.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.2455209Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-128-243.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245595432Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-141-25.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245650633Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245719255Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245803967Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245844668Z host_service_logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.245893299Z host_service_logs/masters/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.24593717Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246008572Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246066843Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246142595Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246257798Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246366581Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246437533Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246508064Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246602537Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246692259Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246786471Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246834422Z ingress_controllers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246845303Z ingress_controllers/default/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246849203Z ingress_controllers/default/router-default-8bdfdcbd8-4fc26/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246880713Z ingress_controllers/default/router-default-8bdfdcbd8-4fc26/haproxy.config [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246982876Z insights-data/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.246992216Z istio/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.247061908Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.247162771Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.247626202Z istio/event-filter.html [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249051277Z istio/timestamp [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249174001Z istio/cluster-scoped-resources/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249187321Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249192151Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249226592Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249320974Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249339505Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249433387Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249501038Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249536939Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.249795686Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.250517264Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.250745769Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.250924384Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.251290953Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.251465198Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.251890208Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.252631986Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.25277571Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.252892823Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.253019416Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.253156359Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.253275012Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocnis.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.253609751Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiorevisions.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.255323133Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiorevisiontags.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.255421226Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istios.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.256831741Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.256944403Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.257055456Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.257170359Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.257311563Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.257496437Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.257844486Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.25803025Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.25839873Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.258538723Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.258667776Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.258867201Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ztunnels.sailoperator.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259391214Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259400525Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259417795Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259480177Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259488487Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259525018Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259694582Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259855266Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259896427Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.259935038Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260038531Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260236205Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260305617Z istio/cluster-scoped-resources/sailoperator.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260314047Z istio/cluster-scoped-resources/sailoperator.io/istiorevisions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260355018Z istio/cluster-scoped-resources/sailoperator.io/istiorevisions/openshift-gateway.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.26042476Z istio/cluster-scoped-resources/sailoperator.io/istios/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.26044487Z istio/cluster-scoped-resources/sailoperator.io/istios/openshift-gateway.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260520092Z istio/namespaces/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260564973Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260615185Z istio/namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260685687Z istio/namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260741888Z istio/namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.26081047Z istio/namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260847111Z istio/namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.260926982Z istio/namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.261781414Z istio/namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.262558253Z istio/namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.262599334Z istio/namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.262641135Z istio/namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.262740987Z istio/namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.262765198Z istio/namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.262857161Z istio/namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.262876141Z istio/namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.262937852Z istio/namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.263016014Z istio/namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.263078006Z istio/namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.263098647Z istio/namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.26324309Z istio/namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.263425204Z istio/namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.266626584Z istio/namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.266728347Z istio/namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.267707561Z istio/namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.267882195Z istio/namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.268228064Z istio/namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.268394908Z istio/namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.268430579Z istio/namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.268600613Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.268608783Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.268662825Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.268773977Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.26887031Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.268913351Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/auth-enabled-test-kserve-route.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269017483Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269110976Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.26928216Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-router-managed-test-llm-4b931143-kserve-route.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269385283Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269499615Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269586908Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.26969416Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269804073Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269847014Z istio/namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269900816Z istio/namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269930726Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.269975377Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270019519Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/auth-enabled-test-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270107361Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270247294Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270376537Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-router-managed-test-llm-4b931143-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270458769Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-router-managed-test-llm-5b1e8f15-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270556102Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/router-with-refs-pd-test-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270639984Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/router-with-refs-test-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270765187Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270786518Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270851919Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/auth-enabled-test-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.270943001Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-fb-opt-125m-with-7ca60146-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271101495Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271227378Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-router-managed-test-llm-4b931143-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271316341Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-router-managed-test-llm-5b1e8f15-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271393713Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/router-with-refs-pd-test-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271485355Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/router-with-refs-test-inference-pool.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271548356Z istio/namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271596407Z istio/namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.27170613Z istio/namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271778172Z istio/namespaces/kserve-ci-e2e-test/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271834153Z istio/namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271879325Z istio/namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271935746Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.271998817Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272030218Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/auth-enabled-test-kserve-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272125361Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/auth-enabled-test-kserve-shadow-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272208813Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/auth-enabled-test-kserve-workload-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272358377Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272467749Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272547631Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272638004Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272771057Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272859809Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.272947971Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-4b931143-kserve-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273037653Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273127236Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273209368Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-5b1e8f15-kserve-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.27332039Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273433173Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273508835Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/router-with-refs-pd-test-kserve-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273600637Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/router-with-refs-pd-test-kserve-workload-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273752321Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/router-with-refs-test-kserve-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273831933Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/router-with-refs-test-kserve-workload-svc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273912265Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/envoyfilters/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.273963526Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/envoyfilters/kuadrant-auth-router-gateway-1.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274053589Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/envoyfilters/kuadrant-auth-router-gateway-2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274140871Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/envoyfilters/kuadrant-router-gateway-1.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274239223Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/envoyfilters/kuadrant-router-gateway-2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274399547Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/envoyfilters/router-gateway-1-authn-ssl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.27449385Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/envoyfilters/router-gateway-2-authn-ssl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274556671Z istio/namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274607353Z istio/namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274825118Z istio/namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274839848Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274854808Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/auth-enabled-test-kserve-85d86d876c-vrqhw.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274974042Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.274986762Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275002682Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275051744Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/current.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275116875Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275194997Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275269439Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275286909Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275293839Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.27532743Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275374532Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275456434Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275526455Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-85d86d876c-vrqhw/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275570196Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275593577Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275774071Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275794552Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275801422Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.275818192Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276201212Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276268114Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276305524Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276312265Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276317985Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276357886Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276459828Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.2765266Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276571001Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276578261Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276588992Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.276634103Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277117975Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277174206Z istio/namespaces/kserve-ci-e2e-test/pods/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277231327Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277284749Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277444953Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277458673Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277462573Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277495214Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/current.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277575486Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277642638Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277771221Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277794612Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277802202Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277809852Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277867423Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.277956525Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278017167Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278073099Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278115809Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278256113Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278264733Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278270924Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278322895Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278456198Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278507369Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.27854602Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278552811Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.27855698Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278593912Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278702394Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278803647Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278841008Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278848198Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.278854018Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.27892605Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.279211567Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.279278898Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schengntw/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.279297069Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.27933149Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.279489114Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.279498504Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.279502064Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.279536145Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280045717Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280115319Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28013223Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28013646Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28015159Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280204972Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280295264Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280372706Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280420667Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280456908Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280616772Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280625502Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280629482Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280668093Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280818007Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280887049Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28093496Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28094302Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28094686Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.280981011Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281082243Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281149125Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281201336Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281217767Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281222487Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281248617Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281484593Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281554475Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schest4hz/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281616356Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281657888Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28175927Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281783851Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281793331Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281833362Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.281942635Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282009666Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-66f88bc44dcnc5x/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282047707Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282086038Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28216971Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28218129Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282200861Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282223151Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282321454Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282385186Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-7c5bd57d44b7wp8/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282455287Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282524479Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282684283Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282698263Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282705103Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282774095Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.282901599Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28296907Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283009091Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283015851Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283021211Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283067583Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283455742Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283523694Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevgktk/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283575455Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283611336Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28376324Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28377393Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283786441Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.283825042Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284011826Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284080718Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284132609Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284139869Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.28414599Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284191571Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284654642Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284746014Z istio/namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8fvcj/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284803996Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.284824456Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/config_dump_istiod.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.285664557Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/config_dump_proxy.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.287313538Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/proxy_stats [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.287450581Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/router-gateway-1-openshift-default-75dcfd69c9-dh6qf.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.287590565Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.287599455Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/istio-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.287606825Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.287648186Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.287928443Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-75dcfd69c9-dh6qf/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.288049906Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.288084687Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/config_dump_istiod.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.288891467Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/config_dump_proxy.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.290397235Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/proxy_stats [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.290486437Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/router-gateway-2-openshift-default-78c98f6f4c-ddrqp.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.29061849Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.29062774Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/istio-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.290634441Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.290642571Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.290801615Z istio/namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-78c98f6f4c-ddrqp/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.290914758Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.290934088Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/router-with-refs-pd-test-kserve-6f78896447-wshh4.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291140553Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291150343Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291154384Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291167644Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291530643Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291599044Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291623065Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291628405Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291631865Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.291693477Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292000294Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292073806Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292105777Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292113937Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292120757Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292168039Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292277472Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292363354Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-6f78896447-wshh4/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292384874Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292439985Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292560828Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292569379Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.292575239Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.29261644Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293061121Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293123413Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293172384Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293180384Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293189934Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293242265Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293336358Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.29341692Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293450451Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293481131Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293647275Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293654616Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293667666Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293742058Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293910552Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.293972474Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294029605Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294037165Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294046956Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294101117Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294209949Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294275611Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294320632Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294328112Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294334032Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294369164Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.29462847Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294728922Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-pd-test-kserve-router-scheduler-5f7487fdfmr99b/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294773023Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294825595Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/router-with-refs-test-kserve-578d595fc-gtvkx.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294949238Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294961088Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.294968748Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295003669Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295569783Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295631885Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295711607Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295722837Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295727397Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295757108Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295861981Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295933102Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-578d595fc-gtvkx/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.295966213Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296007834Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296153508Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296160878Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296164748Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296199179Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296326532Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296399434Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/main/main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296416994Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296423334Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296435755Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296481496Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296575358Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.29664484Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296715942Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296727782Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296734582Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.296771843Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297143122Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297210654Z istio/namespaces/kserve-ci-e2e-test/pods/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76/tokenizer/tokenizer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297260795Z istio/namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297294416Z istio/namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297359758Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297374778Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297421009Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/auth-enabled-test-epp-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297508351Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297590484Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297762468Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/llmisvc-router-managed-test-llm-4b931143-epp-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.29785368Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.297943052Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/router-with-refs-pd-test-epp-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298030224Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/router-with-refs-pd-test-kserve-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298130487Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/router-with-refs-test-epp-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.29823254Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/system:deployers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298331892Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/system:image-builders.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298405424Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/rolebindings/system:image-pullers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298484096Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298591798Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/auth-enabled-test-epp-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298697871Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298801063Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.298944487Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/llmisvc-router-managed-test-llm-4b931143-epp-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299031999Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/llmisvc-router-managed-test-llm-5b1e8f15-epp-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299126912Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/router-with-refs-pd-test-epp-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299210724Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/router-with-refs-pd-test-kserve-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299299546Z istio/namespaces/kserve-ci-e2e-test/rbac.authorization.k8s.io/roles/router-with-refs-test-epp-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299380448Z istio/namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299425109Z istio/namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299526072Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299571263Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299647055Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299697566Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299772318Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299805679Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.299891521Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300115456Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300313101Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300349662Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300397153Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300459375Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300509096Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300583378Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300637599Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300694061Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300778393Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300843524Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.300882965Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.301296456Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.301401818Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.301611263Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.301712956Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302073365Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302213248Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302436304Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302542016Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302553827Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.30269045Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302705211Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302729461Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302822563Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302874685Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302920026Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.302976497Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303051569Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303130881Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303173402Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303220313Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303280445Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303297005Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303331796Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303423958Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303632643Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303753306Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303792758Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303847259Z istio/namespaces/openshift-ingress/openshift-gateway/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.30388242Z istio/namespaces/openshift-ingress/openshift-gateway/debug-syncz.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303962452Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.303971992Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.304012743Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/istiod-openshift-gateway-75c67f8887-qbmcr.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.304130406Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.304145556Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.304149807Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.304195417Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.322902552Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.322957934Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.323027005Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.323063366Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/config_dump_istiod.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.324225535Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/config_dump_proxy.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.326124272Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.326316437Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/proxy_stats [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.32643937Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.32644983Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.326454691Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.326502712Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.32801666Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328296696Z istio/namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328340728Z istio/namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router-default-8bdfdcbd8-4fc26.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.32844872Z istio/namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.32845709Z istio/namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328507052Z istio/namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328560013Z istio/namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328692646Z istio/namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328758518Z istio/namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328779758Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.32882908Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328890571Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328909762Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.328965523Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/rolebindings/istiod-openshift-gateway.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329052515Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329131397Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/rolebindings/system:deployers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329232839Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/rolebindings/system:image-builders.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329295001Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/rolebindings/system:image-pullers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329395694Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329444785Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/roles/istiod-openshift-gateway.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329536787Z istio/namespaces/openshift-ingress/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329584808Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.3296315Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329725912Z istio/namespaces/openshift-operators/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329768883Z istio/namespaces/openshift-operators/openshift-operators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329831345Z istio/namespaces/openshift-operators/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329868665Z istio/namespaces/openshift-operators/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329926187Z istio/namespaces/openshift-operators/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.329972418Z istio/namespaces/openshift-operators/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.3300466Z istio/namespaces/openshift-operators/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330200284Z istio/namespaces/openshift-operators/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330327087Z istio/namespaces/openshift-operators/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330386048Z istio/namespaces/openshift-operators/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330421239Z istio/namespaces/openshift-operators/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33046938Z istio/namespaces/openshift-operators/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330520411Z istio/namespaces/openshift-operators/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330594153Z istio/namespaces/openshift-operators/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330633834Z istio/namespaces/openshift-operators/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330704576Z istio/namespaces/openshift-operators/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330794058Z istio/namespaces/openshift-operators/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33086176Z istio/namespaces/openshift-operators/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.330899491Z istio/namespaces/openshift-operators/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331015254Z istio/namespaces/openshift-operators/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331096776Z istio/namespaces/openshift-operators/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33123796Z istio/namespaces/openshift-operators/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331321742Z istio/namespaces/openshift-operators/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331472115Z istio/namespaces/openshift-operators/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331595418Z istio/namespaces/openshift-operators/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331777343Z istio/namespaces/openshift-operators/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331842915Z istio/namespaces/openshift-operators/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331890616Z istio/namespaces/openshift-operators/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331939347Z istio/namespaces/openshift-operators/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.331990448Z istio/namespaces/openshift-operators/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332012909Z istio/namespaces/openshift-operators/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33206232Z istio/namespaces/openshift-operators/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332146062Z istio/namespaces/openshift-operators/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332300706Z istio/namespaces/openshift-operators/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332344407Z istio/namespaces/openshift-operators/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332389738Z istio/namespaces/openshift-operators/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332428579Z istio/namespaces/openshift-operators/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332464Z istio/namespaces/openshift-operators/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332525672Z istio/namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332535612Z istio/namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.332581473Z istio/namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33287691Z istio/namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.333091226Z istio/namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.333238199Z istio/namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.333449324Z istio/namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.333755902Z istio/namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334033799Z istio/namespaces/openshift-operators/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334041939Z istio/namespaces/openshift-operators/pods/servicemesh-operator3-57f65f65fb-8cd6g/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33408688Z istio/namespaces/openshift-operators/pods/servicemesh-operator3-57f65f65fb-8cd6g/servicemesh-operator3-57f65f65fb-8cd6g.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334203843Z istio/namespaces/openshift-operators/pods/servicemesh-operator3-57f65f65fb-8cd6g/sail-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334215153Z istio/namespaces/openshift-operators/pods/servicemesh-operator3-57f65f65fb-8cd6g/sail-operator/sail-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334228284Z istio/namespaces/openshift-operators/pods/servicemesh-operator3-57f65f65fb-8cd6g/sail-operator/sail-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334272785Z istio/namespaces/openshift-operators/pods/servicemesh-operator3-57f65f65fb-8cd6g/sail-operator/sail-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33446336Z istio/namespaces/openshift-operators/pods/servicemesh-operator3-57f65f65fb-8cd6g/sail-operator/sail-operator/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334510921Z istio/namespaces/openshift-operators/pods/servicemesh-operator3-57f65f65fb-8cd6g/sail-operator/sail-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334557422Z istio/namespaces/openshift-operators/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334600153Z istio/namespaces/openshift-operators/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334657524Z istio/namespaces/openshift-operators/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334725126Z istio/namespaces/openshift-operators/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334781608Z machine_config_ondisk/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334792238Z machine_config_termination_logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334796248Z monitoring/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334799968Z monitoring/alertmanager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334827238Z monitoring/alertmanager/status.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334914771Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.334964032Z monitoring/prometheus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.335005463Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.335093855Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.335155227Z monitoring/prometheus/rules.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.335850084Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.335897115Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.335941096Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.336805688Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.336832868Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.3368921Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.336970672Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.337035264Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.337141566Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.337200008Z monitoring/prometheus/status/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.337247189Z monitoring/prometheus/status/config.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33770333Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.337785972Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.337883485Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338026268Z namespaces/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338065629Z namespaces/cert-manager-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338072379Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338076429Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338163561Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338239413Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338280154Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338327725Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.33848322Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338697715Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.338854879Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339061854Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339294949Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339501575Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339534835Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-nd6tr.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.3397085Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339768281Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339810462Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339858023Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339895064Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.339944466Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340017218Z namespaces/cert-manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340030908Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340036968Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340048988Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340187162Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340360556Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340491429Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340712305Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.340955241Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341193067Z namespaces/default/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341239918Z namespaces/default/default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.34130117Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.34132283Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341396912Z namespaces/default/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341411282Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341523315Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341593137Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341666488Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.34174205Z namespaces/default/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341768381Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341823152Z namespaces/default/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341866564Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341936165Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.341984456Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342014437Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342090819Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342176821Z namespaces/default/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342191372Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342309475Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342389617Z namespaces/default/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342564541Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342631912Z namespaces/default/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342750235Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.342875829Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343027752Z namespaces/default/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343090584Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343138145Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343206317Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343249918Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.3433121Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343361821Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343435463Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343519394Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343574346Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343601136Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343651878Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343800301Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343832772Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343840953Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.343888034Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.344036877Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.344209392Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.344337765Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.34454977Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.344817027Z namespaces/default/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345026692Z namespaces/default/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345066953Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345117514Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345162555Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345230867Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345237547Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345241127Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345282558Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.gateway-api-inference-extension.sigs.k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345386691Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345397051Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345426122Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345535075Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345608426Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345614727Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345651208Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345771681Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345829762Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345862403Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.345955595Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.346000276Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.346010167Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.346038367Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.346213552Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.346403106Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.346533689Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.346778866Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.347071683Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.347324539Z namespaces/kserve/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.34736047Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.347432022Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.347479913Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.347518084Z namespaces/kserve/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.347559085Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.347646787Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.347898823Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348080998Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348331724Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348375565Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348444247Z namespaces/kserve/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348483408Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.34857017Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348649532Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348719774Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348805516Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348845197Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348857397Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.348912268Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.349011121Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.349088793Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.349205126Z namespaces/kserve/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.349247247Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.349484913Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.349590305Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.349946264Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.350018076Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.350273662Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.350386185Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.350635542Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.350781115Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.350824636Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.350918189Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.350955959Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351004481Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351040032Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351124534Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351251747Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351295688Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351328839Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.35138885Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351499853Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351593065Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351637636Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.35176637Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351822831Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351830011Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.351880752Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.352024756Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.3521913Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.352322283Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.352526278Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.352845676Z namespaces/kserve/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353147084Z namespaces/kserve/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353155674Z namespaces/kserve/pods/kserve-controller-manager-6c7654bd99-m8vkw/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353190615Z namespaces/kserve/pods/kserve-controller-manager-6c7654bd99-m8vkw/kserve-controller-manager-6c7654bd99-m8vkw.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353277317Z namespaces/kserve/pods/kserve-controller-manager-6c7654bd99-m8vkw/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353286467Z namespaces/kserve/pods/kserve-controller-manager-6c7654bd99-m8vkw/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353290207Z namespaces/kserve/pods/kserve-controller-manager-6c7654bd99-m8vkw/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353348759Z namespaces/kserve/pods/kserve-controller-manager-6c7654bd99-m8vkw/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353455101Z namespaces/kserve/pods/kserve-controller-manager-6c7654bd99-m8vkw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353519983Z namespaces/kserve/pods/kserve-controller-manager-6c7654bd99-m8vkw/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353568614Z namespaces/kserve/pods/llmisvc-controller-manager-795c469f5-x2dlk/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353584685Z namespaces/kserve/pods/llmisvc-controller-manager-795c469f5-x2dlk/llmisvc-controller-manager-795c469f5-x2dlk.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353691257Z namespaces/kserve/pods/llmisvc-controller-manager-795c469f5-x2dlk/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353710478Z namespaces/kserve/pods/llmisvc-controller-manager-795c469f5-x2dlk/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353717508Z namespaces/kserve/pods/llmisvc-controller-manager-795c469f5-x2dlk/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.353752339Z namespaces/kserve/pods/llmisvc-controller-manager-795c469f5-x2dlk/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.354751754Z namespaces/kserve/pods/llmisvc-controller-manager-795c469f5-x2dlk/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.354817245Z namespaces/kserve/pods/llmisvc-controller-manager-795c469f5-x2dlk/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.354860647Z namespaces/kserve/pods/model-serving-api-fd65d7d6-x2wgq/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.354905348Z namespaces/kserve/pods/model-serving-api-fd65d7d6-x2wgq/model-serving-api-fd65d7d6-x2wgq.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.3549939Z namespaces/kserve/pods/model-serving-api-fd65d7d6-x2wgq/server/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.35500149Z namespaces/kserve/pods/model-serving-api-fd65d7d6-x2wgq/server/server/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.35501649Z namespaces/kserve/pods/model-serving-api-fd65d7d6-x2wgq/server/server/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.355046241Z namespaces/kserve/pods/model-serving-api-fd65d7d6-x2wgq/server/server/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.355135003Z namespaces/kserve/pods/model-serving-api-fd65d7d6-x2wgq/server/server/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.355196885Z namespaces/kserve/pods/model-serving-api-fd65d7d6-x2wgq/server/server/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.355251366Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-z4qqr/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.355290337Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-z4qqr/odh-model-controller-84b54b7f97-z4qqr.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.355365249Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-z4qqr/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.355371499Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-z4qqr/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.355375319Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-z4qqr/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.35540855Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-z4qqr/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.37594867Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-z4qqr/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.375988401Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-z4qqr/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376054613Z namespaces/kserve/pods/s3-init-rpbr4/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376082064Z namespaces/kserve/pods/s3-init-rpbr4/s3-init-rpbr4.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376223477Z namespaces/kserve/pods/s3-init-rpbr4/download-hf-model/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376236907Z namespaces/kserve/pods/s3-init-rpbr4/download-hf-model/download-hf-model/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376244588Z namespaces/kserve/pods/s3-init-rpbr4/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376282409Z namespaces/kserve/pods/s3-init-rpbr4/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376366461Z namespaces/kserve/pods/s3-init-rpbr4/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376429162Z namespaces/kserve/pods/s3-init-rpbr4/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376461553Z namespaces/kserve/pods/s3-init-rpbr4/s3-init/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376480524Z namespaces/kserve/pods/s3-init-rpbr4/s3-init/s3-init/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376499324Z namespaces/kserve/pods/s3-init-rpbr4/s3-init/s3-init/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376535575Z namespaces/kserve/pods/s3-init-rpbr4/s3-init/s3-init/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376840893Z namespaces/kserve/pods/s3-init-rpbr4/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376921344Z namespaces/kserve/pods/s3-init-rpbr4/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376951885Z namespaces/kserve/pods/seaweedfs-64568bcd49-7b68h/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.376980316Z namespaces/kserve/pods/seaweedfs-64568bcd49-7b68h/seaweedfs-64568bcd49-7b68h.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377096099Z namespaces/kserve/pods/seaweedfs-64568bcd49-7b68h/seaweedfs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377107089Z namespaces/kserve/pods/seaweedfs-64568bcd49-7b68h/seaweedfs/seaweedfs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377115659Z namespaces/kserve/pods/seaweedfs-64568bcd49-7b68h/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377167861Z namespaces/kserve/pods/seaweedfs-64568bcd49-7b68h/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377382006Z namespaces/kserve/pods/seaweedfs-64568bcd49-7b68h/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377441547Z namespaces/kserve/pods/seaweedfs-64568bcd49-7b68h/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377463608Z namespaces/kserve/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377514149Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377572601Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377598411Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377662923Z namespaces/kuadrant-system/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377732885Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377792176Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377835227Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.377896289Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.37794104Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378013102Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378287518Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378548405Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378590626Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378627097Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378691879Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.37874855Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378828052Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378864543Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378894893Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.378976636Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379008536Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379043067Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379103339Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379220462Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379324724Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379412306Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379502279Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.37955657Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379784176Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.379940319Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38034779Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.380445022Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38078652Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.380953425Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38118631Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381302153Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381344114Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381476018Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381519309Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381598011Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381635552Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381767645Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381855147Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.381922599Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38197553Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.382060262Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.382111753Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.382178395Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.382185735Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.382225036Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.382446932Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.382692968Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.382866342Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.383121858Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.383383135Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.383707193Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.383769545Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-qjklt.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384060242Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384093283Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384195965Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384293418Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38439212Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384457562Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384513113Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384591655Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384645327Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38479758Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.384938894Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.385050706Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.385147989Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.385155629Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38521049Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino-686db986cb-n5rxl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.385295072Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.385302073Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.385305573Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.385351084Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.387534158Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38761359Z namespaces/kuadrant-system/pods/authorino-686db986cb-n5rxl/authorino/authorino/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.387660851Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.387749933Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/authorino-operator-6d75c86569-cxdzr.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.387861206Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.387869767Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.387879797Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.387934498Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388108833Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388195675Z namespaces/kuadrant-system/pods/authorino-operator-6d75c86569-cxdzr/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388243006Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388302917Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/dns-operator-controller-manager-65b49595d7-knbkx.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38841188Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38842223Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.38843055Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388489962Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388613655Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388730298Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-knbkx/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388776329Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388840341Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin-7dbb555447-w6b6b.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388921203Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388930313Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.388938723Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.389002125Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.389090597Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.389176689Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-7dbb555447-w6b6b/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.3892318Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.389293202Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.389430385Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.389439995Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.389449776Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.389513427Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.397943547Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398005328Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.39805454Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398101781Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador-limitador-69574b596d-qnf8x.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398195993Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398205753Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398210473Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398248974Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398330107Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398388028Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-qnf8x/limitador/limitador/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.39847082Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398525341Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/limitador-operator-controller-manager-6f9f468797-cgn2h.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398612863Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398621974Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398626774Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398661035Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398850189Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398916351Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-cgn2h/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.398967232Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399016413Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399048494Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399088955Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399135796Z namespaces/kube-node-lease/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399142777Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399146427Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399174528Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-128-226.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.39926267Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-128-243.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399344692Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-141-25.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399381333Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399398233Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399452174Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.39965926Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.399884385Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.400026618Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.400225763Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40047909Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.400789037Z namespaces/kube-public/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.400802688Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.400811768Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.400862659Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.401004903Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.401177697Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40130388Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.401495065Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.401749871Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.401990387Z namespaces/kube-system/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402029808Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402140351Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402188942Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402247784Z namespaces/kube-system/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402297695Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402411078Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402484Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402558321Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402613203Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402651104Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402740166Z namespaces/kube-system/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402778097Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402862609Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402922321Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.402958142Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403036924Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403081554Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403098665Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403146446Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-pfrpggv7ka5ip7jqtwmc4fefci.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403232618Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4033178Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403403493Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403559686Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403642908Z namespaces/kube-system/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40369614Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403914405Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.403989347Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.404260194Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.404336466Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.404594092Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.404744746Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405033023Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405088264Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405202807Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405249978Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405288519Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405336561Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405371271Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405446463Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405515205Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405564836Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405604177Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405644438Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40570454Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405755961Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405767471Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.405879964Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.406025427Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.406195802Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.406325345Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.406506699Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.406775416Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.407006442Z namespaces/kube-system/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.407200777Z namespaces/kube-system/pods/global-pull-secret-syncer-hpg5s/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.407236868Z namespaces/kube-system/pods/global-pull-secret-syncer-hpg5s/global-pull-secret-syncer-hpg5s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40731847Z namespaces/kube-system/pods/global-pull-secret-syncer-hpg5s/global-pull-secret-syncer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40732673Z namespaces/kube-system/pods/global-pull-secret-syncer-hpg5s/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40733107Z namespaces/kube-system/pods/global-pull-secret-syncer-hpg5s/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.407352441Z namespaces/kube-system/pods/global-pull-secret-syncer-hpg5s/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40773641Z namespaces/kube-system/pods/global-pull-secret-syncer-hpg5s/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.407872013Z namespaces/kube-system/pods/global-pull-secret-syncer-hpg5s/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.407920705Z namespaces/kube-system/pods/global-pull-secret-syncer-pvj8k/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.407965876Z namespaces/kube-system/pods/global-pull-secret-syncer-pvj8k/global-pull-secret-syncer-pvj8k.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408042878Z namespaces/kube-system/pods/global-pull-secret-syncer-pvj8k/global-pull-secret-syncer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408051198Z namespaces/kube-system/pods/global-pull-secret-syncer-pvj8k/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408056168Z namespaces/kube-system/pods/global-pull-secret-syncer-pvj8k/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408092169Z namespaces/kube-system/pods/global-pull-secret-syncer-pvj8k/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408473388Z namespaces/kube-system/pods/global-pull-secret-syncer-pvj8k/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.40854004Z namespaces/kube-system/pods/global-pull-secret-syncer-pvj8k/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408594122Z namespaces/kube-system/pods/global-pull-secret-syncer-rvr6v/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408635383Z namespaces/kube-system/pods/global-pull-secret-syncer-rvr6v/global-pull-secret-syncer-rvr6v.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408738755Z namespaces/kube-system/pods/global-pull-secret-syncer-rvr6v/global-pull-secret-syncer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408752615Z namespaces/kube-system/pods/global-pull-secret-syncer-rvr6v/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408756806Z namespaces/kube-system/pods/global-pull-secret-syncer-rvr6v/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.408787776Z namespaces/kube-system/pods/global-pull-secret-syncer-rvr6v/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.409129995Z namespaces/kube-system/pods/global-pull-secret-syncer-rvr6v/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.409200426Z namespaces/kube-system/pods/global-pull-secret-syncer-rvr6v/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.409249078Z namespaces/kube-system/pods/konnectivity-agent-6z4c5/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.409291209Z namespaces/kube-system/pods/konnectivity-agent-6z4c5/konnectivity-agent-6z4c5.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.409382371Z namespaces/kube-system/pods/konnectivity-agent-6z4c5/konnectivity-agent/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.409388921Z namespaces/kube-system/pods/konnectivity-agent-6z4c5/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.409394711Z namespaces/kube-system/pods/konnectivity-agent-6z4c5/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.409428162Z namespaces/kube-system/pods/konnectivity-agent-6z4c5/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.410396736Z namespaces/kube-system/pods/konnectivity-agent-6z4c5/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.410462338Z namespaces/kube-system/pods/konnectivity-agent-6z4c5/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.410477478Z namespaces/kube-system/pods/konnectivity-agent-pt4g7/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.41053593Z namespaces/kube-system/pods/konnectivity-agent-pt4g7/konnectivity-agent-pt4g7.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.410604871Z namespaces/kube-system/pods/konnectivity-agent-pt4g7/konnectivity-agent/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.410622302Z namespaces/kube-system/pods/konnectivity-agent-pt4g7/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.410626712Z namespaces/kube-system/pods/konnectivity-agent-pt4g7/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.410634352Z namespaces/kube-system/pods/konnectivity-agent-pt4g7/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.411607626Z namespaces/kube-system/pods/konnectivity-agent-pt4g7/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.411696388Z namespaces/kube-system/pods/konnectivity-agent-pt4g7/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.411726989Z namespaces/kube-system/pods/konnectivity-agent-xv9fc/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.411798211Z namespaces/kube-system/pods/konnectivity-agent-xv9fc/konnectivity-agent-xv9fc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.411860513Z namespaces/kube-system/pods/konnectivity-agent-xv9fc/konnectivity-agent/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.411870933Z namespaces/kube-system/pods/konnectivity-agent-xv9fc/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.411879543Z namespaces/kube-system/pods/konnectivity-agent-xv9fc/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.411914544Z namespaces/kube-system/pods/konnectivity-agent-xv9fc/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.412845377Z namespaces/kube-system/pods/konnectivity-agent-xv9fc/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.412913709Z namespaces/kube-system/pods/konnectivity-agent-xv9fc/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.412933989Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.41297257Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413058042Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal/haproxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413066473Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413077763Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413119144Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413405311Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413470733Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413505033Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413552315Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413632897Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal/haproxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413641187Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413645887Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.413666288Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414035597Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414102698Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-243.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414129499Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.41417939Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414220251Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal/haproxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414238702Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414245892Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414285093Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414496098Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.41456115Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-25.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4145865Z namespaces/kube-system/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414620761Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414702773Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414729124Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414773165Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414868687Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414919248Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.414950709Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415008591Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415038431Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415086583Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415093433Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415109803Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415149384Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415234876Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415290508Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415300848Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415338119Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415528444Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415772Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.415914023Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.416105908Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.416360335Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.416620561Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.416627161Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.416630811Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.416666342Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.416852347Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.417012371Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.417144334Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.417328708Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.417550884Z namespaces/open-cluster-management-d9a74490-1df8-4f47-b68c-b2673f0f1/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.417820171Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.417863952Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.417920973Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.417946994Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418008125Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418041396Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418125488Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.41819815Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418274632Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418325893Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418367754Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418409675Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418448536Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418525798Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418575899Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418616951Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418729473Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418802305Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.418847596Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.419720538Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.41980084Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.419881752Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.419956484Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420032215Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420147429Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420310793Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420367884Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420411275Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420457466Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420496997Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420550528Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.42059701Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420692842Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420785314Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420844426Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420887467Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420947158Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.420985829Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.42103266Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.421080382Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.421116453Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.421126873Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.421156234Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.421295007Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.421464371Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.421595885Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.421853711Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422114047Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422312142Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422349563Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422398874Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422431455Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422509017Z namespaces/openshift-apiserver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422553858Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422778374Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422815685Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422858676Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422902927Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.422984319Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423051201Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423127123Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423168464Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423206705Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423263116Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423303057Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423378529Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4234324Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423472931Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423545823Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423638605Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423668576Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423803929Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423890962Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.423954733Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424031605Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424110337Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4242263Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424384014Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424426995Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424470356Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424521077Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424564668Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.42462605Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.42465623Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424771483Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424843525Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424896427Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424925217Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.424973848Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.425015309Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.425056471Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.425071491Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.425118932Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.425258816Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.42543177Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.425566673Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.425825919Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426052595Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.42626135Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426297291Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426357003Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426396614Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426428774Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426436065Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426442395Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426500936Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426579728Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426587858Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426635Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426822924Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.426989728Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.427122342Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.427297426Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.427524612Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.427757158Z namespaces/openshift-authentication/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.427770758Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.427780908Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.427830529Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.427975593Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.428149337Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.428282551Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.428468625Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.428706911Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.428931997Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.428942667Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.428947497Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.428959097Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.429040579Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4290639Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.42908005Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.429248655Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.429419629Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.429553702Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.429765157Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.429993023Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430201218Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430384823Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430396333Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430422404Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430498055Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430509886Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430523946Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430591798Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430611818Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43065293Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.430833574Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.431002528Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.431132041Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.431324296Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.431547292Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.431824599Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43186834Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.431926561Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.431981292Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.431997743Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432058094Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432138887Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432208358Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43228877Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432310301Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432364342Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432418423Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432437824Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432515726Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432572197Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432592188Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432704281Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432762962Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432772632Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432810943Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432895365Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.432935786Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433029838Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43309844Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433189902Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433254574Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433332336Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43347921Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433585292Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433624923Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433668034Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433808438Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433843839Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43389401Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433926501Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.433975662Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434027203Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434092755Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434165887Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434206568Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434239279Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43429403Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434334891Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434390332Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434403233Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434449544Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434593467Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.434872704Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435006888Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435211753Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435451669Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435660924Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435740836Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435782007Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435789597Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435830648Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435878459Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435923461Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.435961902Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436007963Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436085694Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436139706Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436205008Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436246229Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43629713Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436335301Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436458914Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436532046Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436605827Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436663829Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436737161Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436846713Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.436930315Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437010597Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437065549Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4371083Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437179612Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437223213Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437230043Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437264194Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437347906Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437428548Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43753711Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437627303Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437668044Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.437828828Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43790336Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438223888Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438297489Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438497324Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438631378Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438776471Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438815562Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438874174Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438923615Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.438966706Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439021327Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439067019Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439109469Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.43914923Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439226432Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439302924Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439349626Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439384256Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439430277Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439473249Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4395457Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439556171Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439609772Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.439842308Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.440022902Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.440150926Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44033802Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.440581926Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.440889944Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.440904844Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.440935125Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/aws-ebs-csi-driver-node-hf54t.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441031657Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-driver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441039388Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-driver/csi-driver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441043018Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-driver/csi-driver/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441075729Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441167821Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441236673Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441252033Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-liveness-probe/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441262483Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441271463Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441335135Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441428027Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441491709Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441511809Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-node-driver-registrar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44152514Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44153665Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441602311Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441706004Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441791506Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hf54t/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441846768Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441888879Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/aws-ebs-csi-driver-node-pwcsh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.441974331Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-driver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442028152Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-driver/csi-driver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442038962Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-driver/csi-driver/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442071863Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442158895Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442225927Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442261798Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-liveness-probe/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442268418Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442273828Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442322269Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442406851Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442477623Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442503524Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-node-driver-registrar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442510724Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442521764Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442560065Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442647437Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44274813Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-pwcsh/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442796541Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442839392Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/aws-ebs-csi-driver-node-z9pcq.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442934075Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-driver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442948955Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-driver/csi-driver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442964325Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-driver/csi-driver/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.442993926Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443110329Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443186861Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443218962Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-liveness-probe/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443227262Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443234082Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443291533Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443374135Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443441787Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443482158Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-node-driver-registrar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443492869Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443499479Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44354896Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443637792Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443771945Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z9pcq/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.443904929Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44395555Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444014082Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444021572Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444054123Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444105004Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444157085Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444212906Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444254377Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444313529Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444320169Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444323649Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44436001Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444422501Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444428952Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444468493Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444531734Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444540044Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444588346Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44476794Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.444944965Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.445075268Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.445328624Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4455767Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.445853717Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.445898968Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44595577Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.445992321Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446036352Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446074673Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446197316Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446266687Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446338129Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446385581Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446421801Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446500243Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446529724Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446577185Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446614676Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4467832Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446835452Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446860382Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446942064Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446987275Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.446995916Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.447041397Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.447108728Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.447145179Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.448039411Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.448131494Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.448282928Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.448364559Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.448566864Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.448739529Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.448893333Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.448969285Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449014766Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449071787Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449111648Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44916939Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449269192Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449314643Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449351624Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449430016Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449506198Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4496026Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449644141Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449733004Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449773005Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449845366Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449881877Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.449949119Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.44998602Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.450029521Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.450052012Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.450102103Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.450245726Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.450419591Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.450563694Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.450838101Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451079317Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451274492Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451283192Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5fnpl/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451327033Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5fnpl/tuned-5fnpl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451424496Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5fnpl/tuned/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45157919Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5fnpl/tuned/tuned/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45158654Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5fnpl/tuned/tuned/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451631431Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5fnpl/tuned/tuned/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451767184Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5fnpl/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451830576Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5fnpl/tuned/tuned/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451882467Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jk5fv/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.451909818Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jk5fv/tuned-jk5fv.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45199583Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jk5fv/tuned/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45200236Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jk5fv/tuned/tuned/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45201072Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jk5fv/tuned/tuned/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452060591Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jk5fv/tuned/tuned/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452161294Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jk5fv/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452224286Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jk5fv/tuned/tuned/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452270087Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zzm2s/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452308897Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zzm2s/tuned-zzm2s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45238855Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zzm2s/tuned/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45239678Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zzm2s/tuned/tuned/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45240189Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zzm2s/tuned/tuned/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452447351Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zzm2s/tuned/tuned/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452551583Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zzm2s/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452627005Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zzm2s/tuned/tuned/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452687607Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452739748Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452779279Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45282399Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452870102Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452884882Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.452935543Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-128-226.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453027335Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-128-243.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453104657Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-141-25.ec2.internal.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453153298Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453193899Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453257031Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453305602Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453361634Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453403095Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453458956Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453500037Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453579839Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453752883Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453853206Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453894337Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453936208Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.453986079Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454033671Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454102392Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454149933Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454194744Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454266696Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454326908Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454361819Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454536653Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454612305Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454776969Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454851941Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.454972294Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455087737Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455258201Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455320652Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455366324Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455428775Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455477226Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455522157Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455570139Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45564306Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455749483Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455805275Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455848145Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455904077Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.455990469Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456080361Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456122472Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456175664Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456219825Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456274396Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456285426Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456329828Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456473341Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456642335Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.456806949Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457000294Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4572414Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457440775Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457456136Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457505897Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator-9547488fd-krhb7.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45762911Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator-watch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45764057Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45764736Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457657081Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457885406Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457958148Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457987299Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457994459Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.457999419Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45803963Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.458756538Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.458866271Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9547488fd-krhb7/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.458919012Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.458947182Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459014614Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459031135Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459123417Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459164278Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459226509Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.45924919Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459313322Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459342082Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459434955Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459535167Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459627879Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459706772Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459738962Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459798644Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459830524Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459911726Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459949998Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.459989819Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460069021Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460123042Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460135112Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460167773Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460259095Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460337657Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460378078Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460483771Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460562713Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460923992Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.460999793Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461119247Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46125529Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461389533Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461456825Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461494916Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461553217Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461598498Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461666Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461789473Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461840414Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.461902436Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462002148Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462092841Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462163492Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462213864Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462296836Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462345107Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46244824Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462504511Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462574763Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462627584Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462769028Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462787648Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.462842169Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.463000583Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.463184898Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.463335062Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.463543747Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.463820414Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464038399Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464045429Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-bd4ccd799-2556x/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46410219Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-bd4ccd799-2556x/volume-data-source-validator-bd4ccd799-2556x.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464214083Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-bd4ccd799-2556x/volume-data-source-validator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464227484Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-bd4ccd799-2556x/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464240474Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-bd4ccd799-2556x/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464280265Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-bd4ccd799-2556x/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464397748Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-bd4ccd799-2556x/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4644891Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-bd4ccd799-2556x/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464532301Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464588653Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464654884Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464663825Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464738177Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464828409Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4648823Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.464949322Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465002853Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465076945Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465139526Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465257439Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46530734Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465380192Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465427803Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465525866Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465619068Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465739201Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465802023Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465861054Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465927286Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.465986557Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46608315Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466145931Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466206713Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466367777Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466441159Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466447499Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46649628Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466567082Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466615143Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466769287Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466862829Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.466977372Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467069194Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467160097Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46730294Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467499565Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467572557Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467631648Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467740561Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467778942Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467854464Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467905435Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.467998057Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46809335Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.468159211Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.468224353Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.468287574Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.468344476Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.468413928Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.468424718Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46848619Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.468638373Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.468944361Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.469105345Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.46932161Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.469568537Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.469814023Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.469860694Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.469927575Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.469984597Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470146221Z namespaces/openshift-config-managed/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470203962Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470282164Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470329665Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470405807Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470452778Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470554791Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470643333Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470776247Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470839748Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4709055Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.470977532Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.471032803Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.471125975Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.471182437Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.471241548Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.47133266Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.471422812Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.471476844Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.474643232Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.474797946Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.474907319Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475008292Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475103794Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475271918Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475393351Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475466803Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475521624Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475631887Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475780471Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475824612Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475892654Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.475938905Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476020027Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476071758Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476180131Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476271403Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476333374Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476394426Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476456688Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476501609Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.47656554Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476595101Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476653833Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.476871038Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.477072373Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.477217556Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.477434002Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.477714239Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478105129Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478146859Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478231592Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478243602Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478277003Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478357865Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478406306Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478476128Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478516209Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.47857671Z namespaces/openshift-config-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.47858521Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478595931Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478663003Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478783675Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478802766Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.478845887Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.479003891Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.479182495Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.479322209Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.479525164Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.479797391Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480049067Z namespaces/openshift-config/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480093488Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480205651Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480235662Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480317544Z namespaces/openshift-config/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480370945Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480475637Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.4805819Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480708353Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480778115Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480846517Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.480898748Z namespaces/openshift-config/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481032191Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481128684Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481187825Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481244887Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481347379Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481420751Z namespaces/openshift-config/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481481082Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481609985Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481723088Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481826371Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.481922003Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482013335Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48217917Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482338534Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482409845Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482472727Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482538989Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48259903Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482685632Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482823556Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482872457Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.482919818Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48299484Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483070542Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483113443Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483156224Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483205805Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483236236Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483284077Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483293358Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483323318Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483470022Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.483638676Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48381254Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484017216Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484258482Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484457126Z namespaces/openshift-config/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484495067Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484540169Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484582179Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484659761Z namespaces/openshift-console-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484734663Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484849166Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484900187Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.484969999Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48500542Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485086652Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485200375Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485298037Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485341998Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485379169Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485440761Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485489812Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485559534Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485618015Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485660886Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485775129Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48581928Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48582582Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485869501Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485938333Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.485978984Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.486535378Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48661756Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.486885257Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.486958738Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487079481Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487200975Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487383279Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487499522Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487540113Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487593904Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487634635Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487686027Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487754338Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48783503Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487913792Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487963923Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.487994804Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488056286Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488093407Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488143278Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488186089Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.48824094Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488277581Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488317382Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488329163Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488375874Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488523987Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488726142Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.488867006Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489067211Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489369258Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489579564Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489588894Z namespaces/openshift-console-operator/pods/console-operator-5dbbb5d744-ktjgp/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489627955Z namespaces/openshift-console-operator/pods/console-operator-5dbbb5d744-ktjgp/console-operator-5dbbb5d744-ktjgp.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489746908Z namespaces/openshift-console-operator/pods/console-operator-5dbbb5d744-ktjgp/console-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489760088Z namespaces/openshift-console-operator/pods/console-operator-5dbbb5d744-ktjgp/console-operator/console-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489766548Z namespaces/openshift-console-operator/pods/console-operator-5dbbb5d744-ktjgp/console-operator/console-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.489803499Z namespaces/openshift-console-operator/pods/console-operator-5dbbb5d744-ktjgp/console-operator/console-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490209379Z namespaces/openshift-console-operator/pods/console-operator-5dbbb5d744-ktjgp/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49024964Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490292271Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490329702Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490366533Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490474906Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490488956Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490496926Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490540358Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490725132Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.490909277Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49104532Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.491238115Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.491475761Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.491749777Z namespaces/openshift-console/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.491799719Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49187216Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.491914081Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.491959133Z namespaces/openshift-console/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.491998554Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.492068116Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49223298Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.492483306Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.492621829Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49264784Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.492733302Z namespaces/openshift-console/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.492762363Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.492847305Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.492888986Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.492931237Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.493003369Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49306025Z namespaces/openshift-console/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.493102141Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.493696956Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.493784128Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494023564Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494101166Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49427655Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494388113Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494569647Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494627579Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494693531Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494763482Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494805144Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494866395Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494906256Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.494986538Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49506012Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495112121Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495151182Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495211743Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495253835Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495300246Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495338197Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495383668Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495391998Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495431079Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495577052Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495782298Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.495916941Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496098905Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496321461Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496536806Z namespaces/openshift-console/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496546127Z namespaces/openshift-console/pods/console-6db995c7bf-t6hv2/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496575497Z namespaces/openshift-console/pods/console-6db995c7bf-t6hv2/console-6db995c7bf-t6hv2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49668956Z namespaces/openshift-console/pods/console-6db995c7bf-t6hv2/console/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496703031Z namespaces/openshift-console/pods/console-6db995c7bf-t6hv2/console/console/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496712931Z namespaces/openshift-console/pods/console-6db995c7bf-t6hv2/console/console/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496766762Z namespaces/openshift-console/pods/console-6db995c7bf-t6hv2/console/console/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496871075Z namespaces/openshift-console/pods/console-6db995c7bf-t6hv2/console/console/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496935946Z namespaces/openshift-console/pods/console-6db995c7bf-t6hv2/console/console/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.496984467Z namespaces/openshift-console/pods/downloads-694c955b9-nvblt/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497022908Z namespaces/openshift-console/pods/downloads-694c955b9-nvblt/downloads-694c955b9-nvblt.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497119471Z namespaces/openshift-console/pods/downloads-694c955b9-nvblt/download-server/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497131361Z namespaces/openshift-console/pods/downloads-694c955b9-nvblt/download-server/download-server/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497137601Z namespaces/openshift-console/pods/downloads-694c955b9-nvblt/download-server/download-server/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497169332Z namespaces/openshift-console/pods/downloads-694c955b9-nvblt/download-server/download-server/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497541992Z namespaces/openshift-console/pods/downloads-694c955b9-nvblt/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497605653Z namespaces/openshift-console/pods/downloads-694c955b9-nvblt/download-server/download-server/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497654164Z namespaces/openshift-console/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497707716Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497795268Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497827719Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49788949Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497933541Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.497997163Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498037294Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498100705Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498138166Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498222038Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49829434Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498375852Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498421873Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498464924Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498516416Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498560567Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498633708Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498712841Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498777272Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498865534Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498928946Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.498969827Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499073599Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499148651Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499295385Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499373627Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499449189Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499563492Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499758567Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499812888Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499854629Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.49989765Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.499937861Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500011743Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500063294Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500131296Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500207808Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500258209Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50029338Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500340681Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500372712Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500468584Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500481114Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500521895Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50068399Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.500867914Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.501002347Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.501194542Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.501438118Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.501636753Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.501696135Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.501754326Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.501795007Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.501870089Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5019081Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502064744Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502103025Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502176456Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502211137Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502287229Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502361081Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502435633Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502481224Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502518435Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502577917Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502617498Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502735841Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502794362Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502839513Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502917595Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502961006Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.502973576Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503014557Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503080009Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50311822Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503236443Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503308805Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503394847Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503468039Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503543741Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503658264Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503870209Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50392311Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.503971201Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504022542Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504057483Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504117975Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504157456Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504237028Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50431154Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504365681Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504394962Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504455693Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504495394Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504548446Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504555076Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504598697Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504780161Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.504949745Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.505079599Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.505267303Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.505499189Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.505735945Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.505813397Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.505874708Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50592571Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.505975931Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506019022Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506123525Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506168036Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506220727Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506263878Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50634098Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506414842Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506487354Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506537825Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506581106Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506633057Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506699559Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506795001Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506849943Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506899304Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.506971006Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507026507Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507070758Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50716035Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507239722Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507312584Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507393556Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507463378Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507580351Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507821147Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507883658Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507917739Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.507984441Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508021372Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508076163Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508110144Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508191456Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508276248Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50833892Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508384081Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508429942Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508460433Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508519274Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508554505Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508606716Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508650797Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508716409Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.50873207Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508786401Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.508930304Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.509102539Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.509239372Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.509433737Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.509746405Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510229237Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510274858Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510313359Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51036391Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510402681Z namespaces/openshift-dns/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510434242Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510495393Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510534744Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510599326Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510627757Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510811151Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510879293Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.510969335Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511001396Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511041517Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511104318Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511139859Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511218061Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511271673Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511318454Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511391476Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511445537Z namespaces/openshift-dns/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511491088Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511585361Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511688683Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.511920219Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5119922Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.512253327Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51237409Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.512541894Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.512609866Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.512654117Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.512734779Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51276618Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.512829831Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.512867612Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.512946144Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513017546Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513078158Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513108198Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51317012Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513197081Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513254272Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513289603Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513333064Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513355144Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513402695Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513541519Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513734044Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.513870967Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.514061382Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.514357099Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.514609845Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.514616596Z namespaces/openshift-dns/pods/dns-default-d97j4/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.514658427Z namespaces/openshift-dns/pods/dns-default-d97j4/dns-default-d97j4.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51478833Z namespaces/openshift-dns/pods/dns-default-d97j4/dns/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51479999Z namespaces/openshift-dns/pods/dns-default-d97j4/dns/dns/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5148034Z namespaces/openshift-dns/pods/dns-default-d97j4/dns/dns/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.514859952Z namespaces/openshift-dns/pods/dns-default-d97j4/dns/dns/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.514986195Z namespaces/openshift-dns/pods/dns-default-d97j4/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515053936Z namespaces/openshift-dns/pods/dns-default-d97j4/dns/dns/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515088087Z namespaces/openshift-dns/pods/dns-default-d97j4/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515098628Z namespaces/openshift-dns/pods/dns-default-d97j4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515104328Z namespaces/openshift-dns/pods/dns-default-d97j4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515110498Z namespaces/openshift-dns/pods/dns-default-d97j4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515223281Z namespaces/openshift-dns/pods/dns-default-d97j4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515322293Z namespaces/openshift-dns/pods/dns-default-d97j4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515364494Z namespaces/openshift-dns/pods/dns-default-smfqb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515407165Z namespaces/openshift-dns/pods/dns-default-smfqb/dns-default-smfqb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515500747Z namespaces/openshift-dns/pods/dns-default-smfqb/dns/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515507408Z namespaces/openshift-dns/pods/dns-default-smfqb/dns/dns/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515510888Z namespaces/openshift-dns/pods/dns-default-smfqb/dns/dns/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515542019Z namespaces/openshift-dns/pods/dns-default-smfqb/dns/dns/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515633691Z namespaces/openshift-dns/pods/dns-default-smfqb/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515742184Z namespaces/openshift-dns/pods/dns-default-smfqb/dns/dns/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515783675Z namespaces/openshift-dns/pods/dns-default-smfqb/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515792075Z namespaces/openshift-dns/pods/dns-default-smfqb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515800405Z namespaces/openshift-dns/pods/dns-default-smfqb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515841146Z namespaces/openshift-dns/pods/dns-default-smfqb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.515935849Z namespaces/openshift-dns/pods/dns-default-smfqb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5159996Z namespaces/openshift-dns/pods/dns-default-smfqb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516039061Z namespaces/openshift-dns/pods/dns-default-tcnn8/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516082192Z namespaces/openshift-dns/pods/dns-default-tcnn8/dns-default-tcnn8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516158544Z namespaces/openshift-dns/pods/dns-default-tcnn8/dns/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516166124Z namespaces/openshift-dns/pods/dns-default-tcnn8/dns/dns/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516171974Z namespaces/openshift-dns/pods/dns-default-tcnn8/dns/dns/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516215455Z namespaces/openshift-dns/pods/dns-default-tcnn8/dns/dns/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516304378Z namespaces/openshift-dns/pods/dns-default-tcnn8/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516368419Z namespaces/openshift-dns/pods/dns-default-tcnn8/dns/dns/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5164154Z namespaces/openshift-dns/pods/dns-default-tcnn8/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51642163Z namespaces/openshift-dns/pods/dns-default-tcnn8/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516426511Z namespaces/openshift-dns/pods/dns-default-tcnn8/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516468012Z namespaces/openshift-dns/pods/dns-default-tcnn8/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516564214Z namespaces/openshift-dns/pods/dns-default-tcnn8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516623246Z namespaces/openshift-dns/pods/dns-default-tcnn8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516700888Z namespaces/openshift-dns/pods/node-resolver-6znm2/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51680637Z namespaces/openshift-dns/pods/node-resolver-6znm2/node-resolver-6znm2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516891492Z namespaces/openshift-dns/pods/node-resolver-6znm2/dns-node-resolver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516897813Z namespaces/openshift-dns/pods/node-resolver-6znm2/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516901293Z namespaces/openshift-dns/pods/node-resolver-6znm2/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.516937343Z namespaces/openshift-dns/pods/node-resolver-6znm2/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517024956Z namespaces/openshift-dns/pods/node-resolver-6znm2/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517108938Z namespaces/openshift-dns/pods/node-resolver-6znm2/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517154559Z namespaces/openshift-dns/pods/node-resolver-n8bz5/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51719627Z namespaces/openshift-dns/pods/node-resolver-n8bz5/node-resolver-n8bz5.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517285282Z namespaces/openshift-dns/pods/node-resolver-n8bz5/dns-node-resolver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517294802Z namespaces/openshift-dns/pods/node-resolver-n8bz5/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517300782Z namespaces/openshift-dns/pods/node-resolver-n8bz5/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517340813Z namespaces/openshift-dns/pods/node-resolver-n8bz5/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517425166Z namespaces/openshift-dns/pods/node-resolver-n8bz5/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517497267Z namespaces/openshift-dns/pods/node-resolver-n8bz5/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517535318Z namespaces/openshift-dns/pods/node-resolver-zqzpb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517578469Z namespaces/openshift-dns/pods/node-resolver-zqzpb/node-resolver-zqzpb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517653521Z namespaces/openshift-dns/pods/node-resolver-zqzpb/dns-node-resolver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517660051Z namespaces/openshift-dns/pods/node-resolver-zqzpb/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517665272Z namespaces/openshift-dns/pods/node-resolver-zqzpb/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517757844Z namespaces/openshift-dns/pods/node-resolver-zqzpb/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517839146Z namespaces/openshift-dns/pods/node-resolver-zqzpb/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517908898Z namespaces/openshift-dns/pods/node-resolver-zqzpb/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517944288Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.517981049Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518037551Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518078652Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518146633Z namespaces/openshift-etcd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518189894Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518245016Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518280767Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518337778Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518378889Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518456471Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518529273Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518612995Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518651986Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518712518Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518769169Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.51881183Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518882152Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518946203Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.518989374Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519066726Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519131688Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519178959Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519266491Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519342483Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519417945Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519501407Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519568749Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519712962Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519874696Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519921928Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.519962898Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52000563Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52004534Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520087461Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520126673Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520208135Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520284026Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520331038Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520373229Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52042452Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520470371Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520518012Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520530132Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520581094Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.520750918Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.521004934Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.521150988Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.521338553Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.521572709Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.521925127Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.521954288Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52203305Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522067541Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522147283Z namespaces/openshift-host-network/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522202124Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522264796Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522311257Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522363698Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522411209Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522485441Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522563133Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522649425Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522736918Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522779188Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52284762Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.522914742Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523004604Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523047655Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523082346Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523160198Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52323278Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523281611Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523378923Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523455385Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523534147Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523610839Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523738202Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523888516Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.523988909Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52404487Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524088071Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524148993Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524190663Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524242335Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524284926Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524342107Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524377978Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52445263Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524531742Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524592053Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524625834Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524703496Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524799409Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52485659Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52486999Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.524904741Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.525054195Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.525225119Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.525363713Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.525555017Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.525815714Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526031209Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52606841Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526118111Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526153532Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526246995Z namespaces/openshift-image-registry/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526293246Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526372378Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526399669Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52646386Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526494281Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526596213Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526776928Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526961203Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.526995473Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527038984Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527083495Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527120766Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.52725321Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527312051Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527345292Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527424664Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527470295Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527476655Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527515026Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527563157Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.527597068Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.528505621Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.528621124Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5289015Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.528989093Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529200268Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529325431Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529517316Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529569617Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529604288Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529667579Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529739061Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529786273Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529815763Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529898545Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.529973777Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530025099Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530060119Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530122841Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530175822Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530268934Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530352996Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530397648Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530446949Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.53048667Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530541061Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530582922Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530591613Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530625633Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.530829208Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.531022633Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.531162757Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.531371402Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.531620978Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.531855194Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.531865104Z namespaces/openshift-image-registry/pods/image-registry-749ffdbb68-qbnt2/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.531889235Z namespaces/openshift-image-registry/pods/image-registry-749ffdbb68-qbnt2/image-registry-749ffdbb68-qbnt2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.532000748Z namespaces/openshift-image-registry/pods/image-registry-749ffdbb68-qbnt2/registry/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.532007598Z namespaces/openshift-image-registry/pods/image-registry-749ffdbb68-qbnt2/registry/registry/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.532010898Z namespaces/openshift-image-registry/pods/image-registry-749ffdbb68-qbnt2/registry/registry/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.532057849Z namespaces/openshift-image-registry/pods/image-registry-749ffdbb68-qbnt2/registry/registry/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534443458Z namespaces/openshift-image-registry/pods/image-registry-749ffdbb68-qbnt2/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534478509Z namespaces/openshift-image-registry/pods/image-registry-749ffdbb68-qbnt2/registry/registry/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534531711Z namespaces/openshift-image-registry/pods/node-ca-frzbm/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534575181Z namespaces/openshift-image-registry/pods/node-ca-frzbm/node-ca-frzbm.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534653783Z namespaces/openshift-image-registry/pods/node-ca-frzbm/node-ca/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534667014Z namespaces/openshift-image-registry/pods/node-ca-frzbm/node-ca/node-ca/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534704885Z namespaces/openshift-image-registry/pods/node-ca-frzbm/node-ca/node-ca/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534740406Z namespaces/openshift-image-registry/pods/node-ca-frzbm/node-ca/node-ca/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.53493701Z namespaces/openshift-image-registry/pods/node-ca-frzbm/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.534996932Z namespaces/openshift-image-registry/pods/node-ca-frzbm/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535028223Z namespaces/openshift-image-registry/pods/node-ca-g28pn/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535069614Z namespaces/openshift-image-registry/pods/node-ca-g28pn/node-ca-g28pn.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535148926Z namespaces/openshift-image-registry/pods/node-ca-g28pn/node-ca/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535168156Z namespaces/openshift-image-registry/pods/node-ca-g28pn/node-ca/node-ca/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535173987Z namespaces/openshift-image-registry/pods/node-ca-g28pn/node-ca/node-ca/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535185557Z namespaces/openshift-image-registry/pods/node-ca-g28pn/node-ca/node-ca/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535354011Z namespaces/openshift-image-registry/pods/node-ca-g28pn/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535417853Z namespaces/openshift-image-registry/pods/node-ca-g28pn/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535468054Z namespaces/openshift-image-registry/pods/node-ca-hw222/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535516175Z namespaces/openshift-image-registry/pods/node-ca-hw222/node-ca-hw222.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535574897Z namespaces/openshift-image-registry/pods/node-ca-hw222/node-ca/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535581517Z namespaces/openshift-image-registry/pods/node-ca-hw222/node-ca/node-ca/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535586467Z namespaces/openshift-image-registry/pods/node-ca-hw222/node-ca/node-ca/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535616898Z namespaces/openshift-image-registry/pods/node-ca-hw222/node-ca/node-ca/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535815083Z namespaces/openshift-image-registry/pods/node-ca-hw222/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535875034Z namespaces/openshift-image-registry/pods/node-ca-hw222/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535914325Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.535949776Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.536008897Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.536045348Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.536089569Z namespaces/openshift-infra/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.53609667Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.53610873Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.536139811Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.536286124Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.536459019Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.536596552Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.536820137Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537057353Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537276179Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.53731601Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537373321Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537405742Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537462113Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537491364Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537598127Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537692649Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537792141Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537834153Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537867663Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537910895Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.537950796Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538036538Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538076628Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.53812311Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538203562Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538263643Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538312885Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538432937Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538506019Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538654853Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538763866Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.538924909Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539113424Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539277998Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.53933489Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539368841Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539426722Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539457483Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539508074Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539549895Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539625117Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.53973777Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539776241Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539816122Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539860303Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539906144Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539944965Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.539954605Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.540001626Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54013239Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.540308394Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.540444347Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.540650903Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541000911Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541210837Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541219037Z namespaces/openshift-ingress-canary/pods/ingress-canary-8d8zh/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541241697Z namespaces/openshift-ingress-canary/pods/ingress-canary-8d8zh/ingress-canary-8d8zh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541328789Z namespaces/openshift-ingress-canary/pods/ingress-canary-8d8zh/serve-healthcheck-canary/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5413362Z namespaces/openshift-ingress-canary/pods/ingress-canary-8d8zh/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54134046Z namespaces/openshift-ingress-canary/pods/ingress-canary-8d8zh/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541373941Z namespaces/openshift-ingress-canary/pods/ingress-canary-8d8zh/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541475033Z namespaces/openshift-ingress-canary/pods/ingress-canary-8d8zh/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541534485Z namespaces/openshift-ingress-canary/pods/ingress-canary-8d8zh/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541574846Z namespaces/openshift-ingress-canary/pods/ingress-canary-9gk6x/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541615347Z namespaces/openshift-ingress-canary/pods/ingress-canary-9gk6x/ingress-canary-9gk6x.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541695618Z namespaces/openshift-ingress-canary/pods/ingress-canary-9gk6x/serve-healthcheck-canary/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541714459Z namespaces/openshift-ingress-canary/pods/ingress-canary-9gk6x/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541718909Z namespaces/openshift-ingress-canary/pods/ingress-canary-9gk6x/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54176785Z namespaces/openshift-ingress-canary/pods/ingress-canary-9gk6x/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541863973Z namespaces/openshift-ingress-canary/pods/ingress-canary-9gk6x/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541928074Z namespaces/openshift-ingress-canary/pods/ingress-canary-9gk6x/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.541965015Z namespaces/openshift-ingress-canary/pods/ingress-canary-fmtjs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542008506Z namespaces/openshift-ingress-canary/pods/ingress-canary-fmtjs/ingress-canary-fmtjs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542081208Z namespaces/openshift-ingress-canary/pods/ingress-canary-fmtjs/serve-healthcheck-canary/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542090588Z namespaces/openshift-ingress-canary/pods/ingress-canary-fmtjs/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542099719Z namespaces/openshift-ingress-canary/pods/ingress-canary-fmtjs/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542131149Z namespaces/openshift-ingress-canary/pods/ingress-canary-fmtjs/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542223972Z namespaces/openshift-ingress-canary/pods/ingress-canary-fmtjs/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542289603Z namespaces/openshift-ingress-canary/pods/ingress-canary-fmtjs/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542317284Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542358865Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542409986Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542445027Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542517069Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54255858Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542622012Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542658482Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542739755Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542776585Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.542857267Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54297368Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543048722Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543092923Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543136234Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543179625Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543227336Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543300348Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543346979Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54338269Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543463612Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543515364Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.543550975Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.544435806Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.544516399Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.544635482Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.544744404Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.544816466Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.544936209Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545114293Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545172195Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545214286Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545263447Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545303418Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545353779Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54536016Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545403661Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545466862Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545507273Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545578555Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545753069Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545805591Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545849942Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545904443Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.545944694Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546010286Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546054007Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546103928Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546145419Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54619071Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54619777Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546232421Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546303383Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546311503Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546354684Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546549089Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546769564Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.546907808Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.547113233Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.547366169Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.547652046Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.547712658Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.547774189Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54780792Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.547879382Z namespaces/openshift-ingress/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.547916943Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.547984035Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548012755Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548064747Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548104228Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.54818351Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548412625Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548628941Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548693902Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548797845Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548861587Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548887117Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.548970659Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549023251Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549060221Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549134143Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549179754Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549189355Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549218975Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549306298Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549383059Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549483882Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549529733Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.549964844Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.550069357Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.550280722Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.550356954Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.55060613Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.550755504Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.550943548Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551041421Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551087712Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551175574Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551184714Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551225155Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551292517Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551332778Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551381429Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.55141678Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551496432Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551572134Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551622595Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551662506Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551737798Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551774579Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5518248Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551868461Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551908692Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551921432Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.551967344Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.552120668Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.552293102Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.552427085Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.552660991Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.552924477Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.553141143Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.553153063Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.553162203Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/istiod-openshift-gateway-75c67f8887-qbmcr.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.553287446Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.553295597Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.553307237Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.553336988Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.573053188Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.573093458Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-75c67f8887-qbmcr/discovery/discovery/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.57314013Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.573197771Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.573348595Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.573363775Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.573369405Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.573383986Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.574898414Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-56ccbcff6d-t7hbd/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.57515354Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575192561Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router-default-8bdfdcbd8-4fc26.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575335254Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575349735Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575356045Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575389286Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575500268Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.57556796Z namespaces/openshift-ingress/pods/router-default-8bdfdcbd8-4fc26/router/router/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575609991Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575645052Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575720084Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575763055Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575831167Z namespaces/openshift-insights/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.575882128Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.57594428Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.57598491Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576044162Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576094153Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576201756Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576310998Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576410961Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576457022Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576499523Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576572775Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576612166Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576666557Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576729349Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576809911Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576853532Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576891093Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.576969535Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.577026756Z namespaces/openshift-insights/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.577067587Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.577936599Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.578024941Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.578312428Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.578432851Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.578753249Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.578836121Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.578984385Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579045726Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579082758Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579147139Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.57917922Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579268102Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579309813Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579353254Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579396295Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579443236Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579502388Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579548439Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.57959064Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579625101Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579736654Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579815006Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579856477Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579899988Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.579957569Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580003Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580055642Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580103633Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580151824Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580197825Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580237346Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580246366Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580291097Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580476012Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580693778Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.580841511Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.581038756Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.581293022Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.581585249Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58159313Z namespaces/openshift-insights/pods/insights-operator-5bbd86d6bd-4bc2h/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.581646181Z namespaces/openshift-insights/pods/insights-operator-5bbd86d6bd-4bc2h/insights-operator-5bbd86d6bd-4bc2h.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.581762024Z namespaces/openshift-insights/pods/insights-operator-5bbd86d6bd-4bc2h/insights-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.581775724Z namespaces/openshift-insights/pods/insights-operator-5bbd86d6bd-4bc2h/insights-operator/insights-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.581779424Z namespaces/openshift-insights/pods/insights-operator-5bbd86d6bd-4bc2h/insights-operator/insights-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.581825675Z namespaces/openshift-insights/pods/insights-operator-5bbd86d6bd-4bc2h/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582202335Z namespaces/openshift-insights/pods/insights-operator-5bbd86d6bd-4bc2h/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582528373Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582570394Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/insights-runtime-extractor-bx7hl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582665226Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582707207Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/exporter/exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582728658Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/exporter/exporter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582760849Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/exporter/exporter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582849591Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582928893Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/exporter/exporter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582967144Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/extractor/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582979584Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/extractor/extractor/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.582985874Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/extractor/extractor/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583040146Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/extractor/extractor/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583134788Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58320043Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/extractor/extractor/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583239251Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583245681Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583249181Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583278702Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583366264Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583434915Z namespaces/openshift-insights/pods/insights-runtime-extractor-bx7hl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583473136Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583513897Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/insights-runtime-extractor-mrjn4.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58360242Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58361223Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/exporter/exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58361579Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/exporter/exporter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583646461Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/exporter/exporter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583751663Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583812805Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/exporter/exporter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583851646Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/extractor/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583858376Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/extractor/extractor/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583867376Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/extractor/extractor/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583897157Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/extractor/extractor/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.583988609Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584058101Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/extractor/extractor/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584093142Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584100192Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584106262Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584138373Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584222825Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584349238Z namespaces/openshift-insights/pods/insights-runtime-extractor-mrjn4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584389239Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58443216Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/insights-runtime-extractor-sfxtc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584514472Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584523993Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/exporter/exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584529823Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/exporter/exporter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584561494Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/exporter/exporter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584650796Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584740358Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/exporter/exporter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584771239Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/extractor/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584779449Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/extractor/extractor/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584782969Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/extractor/extractor/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584879842Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/extractor/extractor/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.584968234Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585038785Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/extractor/extractor/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585074446Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585081596Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585085067Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585118877Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585203559Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585272821Z namespaces/openshift-insights/pods/insights-runtime-extractor-sfxtc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585308052Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585345233Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585400084Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585437715Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585483566Z namespaces/openshift-keda/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585492877Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585496567Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585526858Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58562047Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585690511Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585709022Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585742133Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585806925Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585848106Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.585932817Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5860147Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.586071081Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.586100332Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.586148223Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.586291806Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.586464301Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.18.1-2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.586639325Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.586905262Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587041985Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58724037Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587455505Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587473416Z namespaces/openshift-keda/operators.coreos.com/installplans/install-8bpdl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587569628Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587606299Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.18.1-2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58766228Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587717192Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587768493Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587813074Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587891476Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587932657Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.587994989Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58803127Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588089541Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588106662Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588182884Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588260935Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588337777Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588378988Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58842557Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588477891Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588519572Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588592464Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588647015Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588711117Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588793149Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.58883905Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588879031Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.588983773Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589050335Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589141647Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589212069Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589288831Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589408574Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589574208Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589624499Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589695281Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589753332Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589794833Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589838695Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589878046Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.589955717Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59003505Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590087581Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590127572Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590173273Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590211444Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590256765Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590267265Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590305576Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59045415Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590622384Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590790688Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.590989613Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591219679Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591421984Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591465655Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591511886Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591545347Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591612389Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59164565Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591724861Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591762852Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591815174Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591850595Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.591930917Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592009508Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59208204Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592130722Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592169852Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592219484Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592259545Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592335007Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592382198Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592415769Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592498891Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592569153Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592613174Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592730737Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592809208Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.5928812Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.592961432Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593040424Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593159947Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593309801Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593387263Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593427594Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593474405Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593515896Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593568757Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593605928Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593697041Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593794093Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593831564Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593872335Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593936936Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.593984098Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59406967Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594118741Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594165302Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594211393Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594259835Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594304766Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594311176Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594344657Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59449501Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594664685Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.594837339Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595054574Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595304461Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595503365Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595536856Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595591948Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595623118Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59568705Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595748501Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595811323Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595846534Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595902165Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.595938516Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596027548Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59609984Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596174282Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596223713Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596263084Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596321256Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596351116Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596429158Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596474019Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59651388Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596590692Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596644424Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596700465Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596842549Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59691943Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.596993942Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597073204Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597146276Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597268549Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597437814Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597492585Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597522565Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597581657Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597618138Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597687Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597732881Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597814863Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597885085Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597932146Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.597975017Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598020948Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598058799Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59811285Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598156071Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598198562Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598209613Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598253354Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598403878Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598575502Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.598741846Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59893162Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599155146Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599360751Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599405892Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599478644Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599503755Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599580857Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599616928Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599690899Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.59972499Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599789912Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599826143Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599903455Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.599981217Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600058279Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600096269Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60013941Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600190632Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600235673Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600313485Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600358736Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600398567Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600475279Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60052262Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60052924Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600584832Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600632783Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600691984Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600799027Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.600877729Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601142666Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601221727Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601294879Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601424533Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601643098Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60172886Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601768691Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601808792Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601890594Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601928935Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.601974456Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602054048Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60213006Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602181641Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602214132Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602261203Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602306044Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602353325Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602359856Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602403687Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602566101Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602776806Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.602901589Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603092374Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6033333Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603534915Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603571456Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603623797Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603692269Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603760231Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603803742Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603862793Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603902304Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603960365Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.603993846Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604077048Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60415382Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604233512Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604281364Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604311214Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604362846Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604401736Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604478128Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6045295Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60456556Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604642862Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604782836Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604822887Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60493088Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.604999641Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605080633Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605155835Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605231607Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60535834Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605547565Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605593866Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605637217Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605686768Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605734989Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605777961Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605815811Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605893004Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.605974435Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606020897Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606063118Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606106999Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60614707Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606194281Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606206541Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606241412Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606385696Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60656213Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606731604Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.606928599Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607151835Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60735465Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607392851Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607445692Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607481003Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607542694Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607584015Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607647977Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607699969Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60775381Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607794831Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607869993Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.607946315Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608026457Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608065287Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608108629Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60815314Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608190941Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608266083Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608310934Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608349214Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608422956Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608470628Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608510578Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608608181Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608697623Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608782605Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608856127Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.608932129Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609058982Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609243377Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609301698Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609346759Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60939021Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609428211Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609471913Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609509553Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609584055Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609667527Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609726059Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.60976522Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609815571Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609851582Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609899073Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609912483Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.609956874Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.610099028Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.610267122Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.610400645Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6105909Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.610860897Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611060082Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611098963Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611140734Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611182765Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611262067Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611303268Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.61136566Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.61140767Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611472522Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611514713Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611590015Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611717558Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611815061Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611856642Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611895203Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611939384Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.611979665Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612056117Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612113928Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612161299Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612228261Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612281542Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612288772Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612327093Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612393715Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612443456Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612532989Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612618011Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612855446Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.612926938Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613045691Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613159594Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613350629Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613416411Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613466152Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613513373Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613560784Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613614865Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613655736Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613758879Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613838491Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613890452Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613937263Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.613976644Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.614013065Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.614065776Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.614079097Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.614136138Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.614317983Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.614495717Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.614625041Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.614853816Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615087582Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615299397Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615307117Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7c578b78fc-pr6nz/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615361259Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7c578b78fc-pr6nz/kube-storage-version-migrator-operator-7c578b78fc-pr6nz.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615487972Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7c578b78fc-pr6nz/kube-storage-version-migrator-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615502892Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7c578b78fc-pr6nz/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615512823Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7c578b78fc-pr6nz/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615561644Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7c578b78fc-pr6nz/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615689427Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7c578b78fc-pr6nz/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.61581459Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615845151Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615898002Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.615936563Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616001675Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616045836Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616102757Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616136628Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616184519Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6162188Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616341523Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616451006Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616542558Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616589579Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.61662818Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616688752Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616746243Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616826185Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616872366Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616908147Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.616988219Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.61703963Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617080162Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617181134Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617245156Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617359189Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.61743084Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617545373Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617660886Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.61784423Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617892062Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617935393Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.617983754Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618024095Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618062706Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618104607Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618180589Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618257401Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618302212Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618340363Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618382274Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618491266Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618536868Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618549268Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618595929Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.618774773Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.61901277Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619144913Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619330707Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619558773Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619778609Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619790519Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6198351Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/migrator-d4bcd4867-8dt2x.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619919072Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/graceful-termination/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619925942Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/graceful-termination/graceful-termination/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619929322Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.619972173Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620058305Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620122427Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620150968Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/migrator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620156788Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/migrator/migrator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620164648Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/migrator/migrator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620207439Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/migrator/migrator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620298761Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620372163Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d4bcd4867-8dt2x/migrator/migrator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620408884Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620447305Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620494286Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620532017Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620594019Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62064169Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620821744Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620868796Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620928317Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.620965008Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62104652Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621184553Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621301806Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621336757Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621390269Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621427399Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621546813Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621621544Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621689576Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621748398Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621829589Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62186358Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62187231Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621911451Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.621992084Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.622060735Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.622078266Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.622194449Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.622275801Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.622587638Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6226585Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.622877986Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.622994889Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623179103Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623246065Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623292676Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623353607Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623387098Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6234381Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62347973Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623556912Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623635524Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623698996Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623753777Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623818529Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62385829Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623932652Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.623960873Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.624011324Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.624019934Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.624065315Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.624208268Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.624407504Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.624545677Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.624797623Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.624996468Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625217333Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625423039Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62546026Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-9gf6g.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625529021Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625559732Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625612473Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625651724Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625733306Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625765297Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625840439Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625852629Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-pw9bq/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62587144Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-pw9bq/lws-controller-manager-74f47976d9-pw9bq.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625957002Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-pw9bq/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625963332Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-pw9bq/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.625967292Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-pw9bq/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626004423Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-pw9bq/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626109976Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-pw9bq/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626178928Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-pw9bq/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626212838Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-v6qw2/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626253749Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-v6qw2/lws-controller-manager-74f47976d9-v6qw2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626329791Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-v6qw2/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626336161Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-v6qw2/manager/manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626340612Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-v6qw2/manager/manager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626381983Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-v6qw2/manager/manager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626543527Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-v6qw2/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626610338Z namespaces/openshift-lws-operator/pods/lws-controller-manager-74f47976d9-v6qw2/manager/manager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626635059Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-d9kj2/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62669221Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-d9kj2/openshift-lws-operator-fd8ccff4c-d9kj2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626799903Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-d9kj2/openshift-lws-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626811473Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-d9kj2/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626815383Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-d9kj2/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.626852814Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-d9kj2/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627014488Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-d9kj2/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62707768Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-d9kj2/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627113901Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627148882Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627195163Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627233044Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627277615Z namespaces/openshift-machine-api/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627283825Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627289065Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627327946Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6274752Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627641054Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.627805858Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628001893Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628238509Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628529396Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628537446Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628541686Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628578457Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628732061Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628812653Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628824243Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.628856484Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.629009628Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.629177422Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.629302405Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.62951093Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.629783207Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.629980332Z namespaces/openshift-marketplace/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.629989702Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.629994682Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630021463Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630084934Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630092995Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630117606Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630206858Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.63028283Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630360141Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630413233Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630445034Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630592057Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630812093Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.630948876Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631133551Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631359266Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631580012Z namespaces/openshift-monitoring/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631621133Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631717895Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631753026Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631800407Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631836598Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.631978312Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.632393212Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.632762061Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.632977697Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.633018038Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.633069609Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.63310833Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.633188992Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.633226243Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.633261553Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.633339795Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.633395297Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.633438168Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.636451933Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.636590966Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.637663413Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.637773266Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.638636487Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.638802841Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639266923Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639423577Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639465278Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639610411Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639750565Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639806146Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639833967Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639922079Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.639996621Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640045902Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640084043Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640232467Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640265307Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64037791Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640428672Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640508944Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640539244Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640651477Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640839682Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.640922154Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.641197971Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.641329324Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.641453247Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.641547199Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.641650732Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.641772475Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.641842687Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.641885218Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64196518Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642044822Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642163425Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642264107Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642337179Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642481863Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642571175Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642657247Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64278646Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642876672Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.642955414Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.643001455Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.643040967Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.643088018Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.643096318Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.643135439Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.643336464Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.643550549Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.643697253Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.644027761Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.644300418Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.644575525Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.644618096Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64478342Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64479577Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.644873652Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645035006Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645043466Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645048416Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645083497Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64519678Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645260212Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645301443Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645309273Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645313733Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645350314Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645440966Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645500738Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645539819Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645548159Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645552329Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6455892Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645667522Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645784065Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645797995Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645809105Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645831106Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645864937Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.645952519Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646016721Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646054531Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646062832Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646066322Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646108353Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646195085Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646255516Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646281057Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646286737Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646290847Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646329018Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64641566Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646475642Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646514423Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646520533Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646527643Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646559374Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646642026Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646752279Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64678891Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-f9d7df769-jlxrk/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646832711Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-f9d7df769-jlxrk/cluster-monitoring-operator-f9d7df769-jlxrk.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646935663Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-f9d7df769-jlxrk/cluster-monitoring-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646941924Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-f9d7df769-jlxrk/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646950484Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-f9d7df769-jlxrk/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.646991695Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-f9d7df769-jlxrk/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.647784774Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-f9d7df769-jlxrk/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.647854856Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-f9d7df769-jlxrk/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.647900447Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.647931808Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-state-metrics-678bc549d7-9xm6l.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6480343Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648041361Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648045071Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648099732Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648195284Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648271726Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648298667Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-self/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648307287Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648310677Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648356668Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648440261Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648504482Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648544693Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-state-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648552373Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648555784Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648585844Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64882551Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648892142Z namespaces/openshift-monitoring/pods/kube-state-metrics-678bc549d7-9xm6l/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648933143Z namespaces/openshift-monitoring/pods/metrics-server-796477895d-ppfwg/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.648970834Z namespaces/openshift-monitoring/pods/metrics-server-796477895d-ppfwg/metrics-server-796477895d-ppfwg.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649067026Z namespaces/openshift-monitoring/pods/metrics-server-796477895d-ppfwg/metrics-server/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649076796Z namespaces/openshift-monitoring/pods/metrics-server-796477895d-ppfwg/metrics-server/metrics-server/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649084017Z namespaces/openshift-monitoring/pods/metrics-server-796477895d-ppfwg/metrics-server/metrics-server/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649119678Z namespaces/openshift-monitoring/pods/metrics-server-796477895d-ppfwg/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64920909Z namespaces/openshift-monitoring/pods/metrics-server-796477895d-ppfwg/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649269741Z namespaces/openshift-monitoring/pods/metrics-server-796477895d-ppfwg/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649312872Z namespaces/openshift-monitoring/pods/monitoring-plugin-589576dfc7-cwbxl/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649342973Z namespaces/openshift-monitoring/pods/monitoring-plugin-589576dfc7-cwbxl/monitoring-plugin-589576dfc7-cwbxl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649421735Z namespaces/openshift-monitoring/pods/monitoring-plugin-589576dfc7-cwbxl/monitoring-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649428055Z namespaces/openshift-monitoring/pods/monitoring-plugin-589576dfc7-cwbxl/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649431405Z namespaces/openshift-monitoring/pods/monitoring-plugin-589576dfc7-cwbxl/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649464606Z namespaces/openshift-monitoring/pods/monitoring-plugin-589576dfc7-cwbxl/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649556178Z namespaces/openshift-monitoring/pods/monitoring-plugin-589576dfc7-cwbxl/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.64961925Z namespaces/openshift-monitoring/pods/monitoring-plugin-589576dfc7-cwbxl/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649662541Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649759123Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/node-exporter-2k5fq.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649872576Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/init-textfile/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649882826Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/init-textfile/init-textfile/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649890587Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/init-textfile/init-textfile/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.649931428Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650000479Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650067221Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650106312Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650116242Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650121592Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650155383Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650235535Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650296107Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650342638Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/node-exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650351878Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/node-exporter/node-exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650356458Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/node-exporter/node-exporter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650380579Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650488992Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650559843Z namespaces/openshift-monitoring/pods/node-exporter-2k5fq/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650586514Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650625845Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/node-exporter-kxn65.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650755068Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/init-textfile/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650765699Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/init-textfile/init-textfile/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650769348Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/init-textfile/init-textfile/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650800559Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650873421Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650939383Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650978924Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650986904Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.650990384Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651031835Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651129577Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651196959Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6512291Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/node-exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65123589Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/node-exporter/node-exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6512512Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/node-exporter/node-exporter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651281841Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651398894Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651454745Z namespaces/openshift-monitoring/pods/node-exporter-kxn65/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651496186Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651531627Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/node-exporter-t8h8g.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65162061Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/init-textfile/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65162692Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/init-textfile/init-textfile/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6516322Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/init-textfile/init-textfile/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651688661Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651775704Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651837935Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651866866Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651878816Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651882226Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.651919127Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652010199Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652067021Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652107782Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/node-exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652115422Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/node-exporter/node-exporter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652128022Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/node-exporter/node-exporter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652152253Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652253405Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652320957Z namespaces/openshift-monitoring/pods/node-exporter-t8h8g/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652363488Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652400149Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/openshift-state-metrics-6b8d965dd5-dwz9z.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652541623Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652552893Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652560923Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652597884Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652711557Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652784889Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652818759Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-self/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65282541Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65283081Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652866191Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.652953063Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653019034Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653047545Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/openshift-state-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653065696Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653073706Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653101306Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653201489Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65326491Z namespaces/openshift-monitoring/pods/openshift-state-metrics-6b8d965dd5-dwz9z/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653322822Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653396724Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653581598Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653592849Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653599589Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65363826Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653767133Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653838215Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653852165Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653857345Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653881076Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.653934337Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654019469Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654085281Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654128042Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654137242Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654142502Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654182963Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654263575Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654329337Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654360918Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654367008Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654375778Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654416109Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654497551Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654567153Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654601544Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654610354Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654617324Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654662155Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654779288Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6548478Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654880011Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654892861Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654903511Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.654930702Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655618829Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655690081Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655742352Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655758152Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655772043Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655795813Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655898306Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655968468Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.655985188Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656033089Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/prometheus-operator-864fcfc6f9-jdntm.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656123772Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656134032Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656139322Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656163353Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656258315Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656319697Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656358947Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/prometheus-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656367248Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656371958Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65644974Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656747047Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656836739Z namespaces/openshift-monitoring/pods/prometheus-operator-864fcfc6f9-jdntm/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65686039Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5cf667749c-fkznk/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.656911141Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5cf667749c-fkznk/prometheus-operator-admission-webhook-5cf667749c-fkznk.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657053295Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5cf667749c-fkznk/prometheus-operator-admission-webhook/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657064055Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5cf667749c-fkznk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657070835Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5cf667749c-fkznk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657097826Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5cf667749c-fkznk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657187488Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5cf667749c-fkznk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65726123Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5cf667749c-fkznk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657305751Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657341602Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/telemeter-client-6545f694f6-ctq84.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657493186Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657502606Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657506906Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657529796Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657626729Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657784033Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657803513Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/reload/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657810834Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/reload/reload/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657823564Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/reload/reload/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657863095Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/reload/reload/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.657949987Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658019149Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/reload/reload/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65806143Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/telemeter-client/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65807215Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/telemeter-client/telemeter-client/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65808119Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658189693Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658358767Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658426439Z namespaces/openshift-monitoring/pods/telemeter-client-6545f694f6-ctq84/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65847126Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658604183Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/thanos-querier-5fc75ff8df-4h5kr.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658775797Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658785778Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658789208Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658821088Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658897001Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.658977283Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659003423Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-rules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659014543Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659054124Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659098646Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659191608Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659249029Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6592701Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-web/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65927464Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65927867Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659317301Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659436924Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659503945Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659521806Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659529806Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659536256Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659574877Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.65966054Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659755292Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659815063Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/prom-label-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659823724Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659828164Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659859885Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.659944316Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660004198Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660029409Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/thanos-query/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660039699Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/thanos-query/thanos-query/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660043819Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/thanos-query/thanos-query/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66009261Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660193093Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660255164Z namespaces/openshift-monitoring/pods/thanos-querier-5fc75ff8df-4h5kr/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660293555Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660316826Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660366697Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660405668Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66049116Z namespaces/openshift-multus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660533541Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660595633Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660637854Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660715196Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660744696Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.660992653Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661062784Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661177597Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661223928Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661256339Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661410773Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661531606Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661586127Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661630908Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66168979Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661733801Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661806683Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661852374Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661895705Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.661970327Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.662032308Z namespaces/openshift-multus/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.662068929Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.662205993Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.662284134Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.662777867Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66289817Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.663486804Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.663565026Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.663739441Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.663794632Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.663820593Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.663918365Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.663995257Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664076559Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664163171Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664203362Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664288344Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664370906Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664451639Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66449787Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664540681Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664588332Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664635623Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664709295Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664744016Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664795227Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664830988Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66491106Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.664983602Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665026433Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665065444Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665116245Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665157306Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665208617Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665249038Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665291439Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665300349Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665336981Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665482344Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665653328Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.665813182Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666017137Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666251533Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666459168Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666466658Z namespaces/openshift-multus/pods/multus-6zzmq/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666489109Z namespaces/openshift-multus/pods/multus-6zzmq/multus-6zzmq.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666599592Z namespaces/openshift-multus/pods/multus-6zzmq/kube-multus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666606232Z namespaces/openshift-multus/pods/multus-6zzmq/kube-multus/kube-multus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666609702Z namespaces/openshift-multus/pods/multus-6zzmq/kube-multus/kube-multus/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.666646373Z namespaces/openshift-multus/pods/multus-6zzmq/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66733392Z namespaces/openshift-multus/pods/multus-6zzmq/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667398682Z namespaces/openshift-multus/pods/multus-6zzmq/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667452633Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667563296Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/multus-additional-cni-plugins-c5vrl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667795022Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/bond-cni-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667808982Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667814652Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667845543Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667923205Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.667993026Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668019417Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668023497Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/cni-plugins/cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668026687Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668081209Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6681582Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668234872Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668251703Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/egress-router-binary-copy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668260133Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668268083Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668322664Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668394546Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668460758Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668491649Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668500899Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668510219Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66854071Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668620122Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668745825Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668784046Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/routeoverride-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668788346Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668802936Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668837127Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668924339Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.668995321Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669011692Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni-bincopy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669017252Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669024852Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669088814Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669172376Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669237297Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669270498Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669277418Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669308649Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669362881Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669453043Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669516584Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-c5vrl/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669547635Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669591116Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/multus-additional-cni-plugins-fzkqr.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66974133Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/bond-cni-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.66975634Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669760711Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669771541Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669869493Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669933105Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669946965Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669952095Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/cni-plugins/cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669959665Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.669999936Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670079228Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67014458Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670187841Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/egress-router-binary-copy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670198921Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670204291Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670216382Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670296234Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670362375Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670399136Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670406236Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670409636Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670431937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670499829Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67056411Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670599361Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/routeoverride-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670606241Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670609672Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670636562Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670755885Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670818957Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670848297Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni-bincopy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670857208Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670860598Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670892648Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.670974741Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671041602Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671061293Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671065063Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671070103Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671112264Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671199766Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671262728Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fzkqr/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671300249Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671330569Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/multus-additional-cni-plugins-v2kln.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671460413Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/bond-cni-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671466533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671469933Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671499243Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671584116Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671646567Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671698788Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671708919Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/cni-plugins/cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671725439Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67174276Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671831202Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671901664Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671930614Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/egress-router-binary-copy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671939785Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671945535Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.671980306Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672066518Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672135519Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67216679Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67217417Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672190701Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672237842Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672284013Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672353195Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672382305Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/routeoverride-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672419896Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672432517Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672458427Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672542629Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672605431Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672635602Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni-bincopy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672645192Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672648542Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672700523Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672793306Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672859547Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672894198Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672900068Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672903429Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.672934649Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673014761Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673079163Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-v2kln/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673114404Z namespaces/openshift-multus/pods/multus-c2tf2/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673142504Z namespaces/openshift-multus/pods/multus-c2tf2/multus-c2tf2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673236997Z namespaces/openshift-multus/pods/multus-c2tf2/kube-multus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673246277Z namespaces/openshift-multus/pods/multus-c2tf2/kube-multus/kube-multus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673251457Z namespaces/openshift-multus/pods/multus-c2tf2/kube-multus/kube-multus/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673262878Z namespaces/openshift-multus/pods/multus-c2tf2/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673799271Z namespaces/openshift-multus/pods/multus-c2tf2/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673867193Z namespaces/openshift-multus/pods/multus-c2tf2/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673901433Z namespaces/openshift-multus/pods/multus-nt2fx/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.673937994Z namespaces/openshift-multus/pods/multus-nt2fx/multus-nt2fx.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.674022176Z namespaces/openshift-multus/pods/multus-nt2fx/kube-multus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.674030317Z namespaces/openshift-multus/pods/multus-nt2fx/kube-multus/kube-multus/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.674033797Z namespaces/openshift-multus/pods/multus-nt2fx/kube-multus/kube-multus/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.674062167Z namespaces/openshift-multus/pods/multus-nt2fx/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.674651882Z namespaces/openshift-multus/pods/multus-nt2fx/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.674844037Z namespaces/openshift-multus/pods/multus-nt2fx/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.674865987Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.674926259Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/network-metrics-daemon-cbmww.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675017771Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675027061Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675030611Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675058392Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675151914Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675215446Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675242907Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/network-metrics-daemon/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675247317Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675251217Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675287288Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675444202Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675509893Z namespaces/openshift-multus/pods/network-metrics-daemon-cbmww/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675530774Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675577615Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/network-metrics-daemon-jhbs2.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675655407Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675665587Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675688928Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675721448Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675821221Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675874432Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675893443Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/network-metrics-daemon/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675906453Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675915473Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.675954804Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676094468Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67616771Z namespaces/openshift-multus/pods/network-metrics-daemon-jhbs2/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6761837Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676228591Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/network-metrics-daemon-pxmpj.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676315103Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676326354Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676332114Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676352504Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676434876Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676496928Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676534589Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/network-metrics-daemon/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676541059Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676544369Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676571129Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676757644Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676818546Z namespaces/openshift-multus/pods/network-metrics-daemon-pxmpj/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676855297Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676883807Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676946509Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.676958619Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67697818Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677134314Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677179935Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677218236Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677328318Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67739996Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677442031Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677485662Z namespaces/openshift-must-gather-d7tsl/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677492172Z namespaces/openshift-must-gather-d7tsl/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677499623Z namespaces/openshift-must-gather-d7tsl/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677528953Z namespaces/openshift-must-gather-d7tsl/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677693147Z namespaces/openshift-must-gather-d7tsl/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.677874972Z namespaces/openshift-must-gather-d7tsl/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.678000555Z namespaces/openshift-must-gather-d7tsl/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6782007Z namespaces/openshift-must-gather-d7tsl/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.678472937Z namespaces/openshift-must-gather-d7tsl/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.678713013Z namespaces/openshift-network-console/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.678768694Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.678828026Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.678852286Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.678907018Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.678950359Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679031371Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679151844Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679249736Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679277287Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679316648Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67938269Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6794244Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679473882Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679506873Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679578404Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679622336Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679662677Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679764539Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.67981296Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679842621Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.679951544Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680033886Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680163939Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68022573Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680370924Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680493597Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68060778Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680661161Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680793245Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680833475Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680874047Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680920918Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680966339Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.680997319Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681042561Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681084992Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681114433Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681200495Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681271766Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681318558Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681364429Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6814027Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681438641Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681481671Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681488372Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681522252Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681762229Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.681937233Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.682072546Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.682268451Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.682546698Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.682874616Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.682886937Z namespaces/openshift-network-console/pods/networking-console-plugin-867b76cb74-qhtzq/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.682903997Z namespaces/openshift-network-console/pods/networking-console-plugin-867b76cb74-qhtzq/networking-console-plugin-867b76cb74-qhtzq.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.682996339Z namespaces/openshift-network-console/pods/networking-console-plugin-867b76cb74-qhtzq/networking-console-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6830084Z namespaces/openshift-network-console/pods/networking-console-plugin-867b76cb74-qhtzq/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68301218Z namespaces/openshift-network-console/pods/networking-console-plugin-867b76cb74-qhtzq/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68303727Z namespaces/openshift-network-console/pods/networking-console-plugin-867b76cb74-qhtzq/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683119392Z namespaces/openshift-network-console/pods/networking-console-plugin-867b76cb74-qhtzq/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683198484Z namespaces/openshift-network-console/pods/networking-console-plugin-867b76cb74-qhtzq/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683222085Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683267726Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683303527Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683345208Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683385069Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68343334Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683486281Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683523362Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683564813Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683599894Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683723137Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68381912Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683910582Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683950523Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.683990214Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684053966Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684096697Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684150878Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684189249Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6842308Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684268961Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684345273Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684385444Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684424025Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684498017Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684539118Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684576888Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684664781Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684786164Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.684977518Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68504942Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685249655Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685329987Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68543657Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685485851Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685590474Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685642195Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685701067Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685784709Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68582989Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685872741Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685930452Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.685971573Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686012314Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686048805Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686122267Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686196149Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68624574Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686278021Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686328362Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686367953Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686413784Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686449785Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686489156Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686495646Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686535727Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686698441Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.686890696Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687021569Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687238154Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687486911Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687697176Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687708846Z namespaces/openshift-network-diagnostics/pods/network-check-source-6947749cfc-lxqzc/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687753567Z namespaces/openshift-network-diagnostics/pods/network-check-source-6947749cfc-lxqzc/network-check-source-6947749cfc-lxqzc.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687826029Z namespaces/openshift-network-diagnostics/pods/network-check-source-6947749cfc-lxqzc/check-endpoints/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687834219Z namespaces/openshift-network-diagnostics/pods/network-check-source-6947749cfc-lxqzc/check-endpoints/check-endpoints/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687837689Z namespaces/openshift-network-diagnostics/pods/network-check-source-6947749cfc-lxqzc/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68787028Z namespaces/openshift-network-diagnostics/pods/network-check-source-6947749cfc-lxqzc/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.687979483Z namespaces/openshift-network-diagnostics/pods/network-check-source-6947749cfc-lxqzc/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688044245Z namespaces/openshift-network-diagnostics/pods/network-check-source-6947749cfc-lxqzc/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688093726Z namespaces/openshift-network-diagnostics/pods/network-check-target-8n9hm/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688124577Z namespaces/openshift-network-diagnostics/pods/network-check-target-8n9hm/network-check-target-8n9hm.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688206379Z namespaces/openshift-network-diagnostics/pods/network-check-target-8n9hm/network-check-target-container/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688216049Z namespaces/openshift-network-diagnostics/pods/network-check-target-8n9hm/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688220389Z namespaces/openshift-network-diagnostics/pods/network-check-target-8n9hm/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68824835Z namespaces/openshift-network-diagnostics/pods/network-check-target-8n9hm/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688338162Z namespaces/openshift-network-diagnostics/pods/network-check-target-8n9hm/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688402183Z namespaces/openshift-network-diagnostics/pods/network-check-target-8n9hm/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688438595Z namespaces/openshift-network-diagnostics/pods/network-check-target-bcpdg/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688473315Z namespaces/openshift-network-diagnostics/pods/network-check-target-bcpdg/network-check-target-bcpdg.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688550757Z namespaces/openshift-network-diagnostics/pods/network-check-target-bcpdg/network-check-target-container/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688558418Z namespaces/openshift-network-diagnostics/pods/network-check-target-bcpdg/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688570398Z namespaces/openshift-network-diagnostics/pods/network-check-target-bcpdg/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688590698Z namespaces/openshift-network-diagnostics/pods/network-check-target-bcpdg/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6886654Z namespaces/openshift-network-diagnostics/pods/network-check-target-bcpdg/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688763632Z namespaces/openshift-network-diagnostics/pods/network-check-target-bcpdg/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688791913Z namespaces/openshift-network-diagnostics/pods/network-check-target-gbdt4/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688835584Z namespaces/openshift-network-diagnostics/pods/network-check-target-gbdt4/network-check-target-gbdt4.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688907976Z namespaces/openshift-network-diagnostics/pods/network-check-target-gbdt4/network-check-target-container/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688919476Z namespaces/openshift-network-diagnostics/pods/network-check-target-gbdt4/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688926367Z namespaces/openshift-network-diagnostics/pods/network-check-target-gbdt4/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.688934147Z namespaces/openshift-network-diagnostics/pods/network-check-target-gbdt4/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689030349Z namespaces/openshift-network-diagnostics/pods/network-check-target-gbdt4/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689083591Z namespaces/openshift-network-diagnostics/pods/network-check-target-gbdt4/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689128152Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689146092Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689192543Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689201474Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689239734Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689326277Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689363958Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689406839Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6894824Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689530782Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689564323Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689609333Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689652215Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689733497Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689762437Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689812358Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.68984913Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689927892Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.689998753Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690073835Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690117996Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690153257Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690196298Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690233199Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690308361Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690348872Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690387693Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690461505Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690511426Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690522106Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690537427Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690596938Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690629659Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690766562Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690847624Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.690927236Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691001548Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69107275Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691205823Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691306776Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691350067Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691389828Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691437769Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69147459Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691520561Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691560322Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691600753Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691639714Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691737236Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691810958Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691851509Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69189213Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691942431Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.691971742Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.692018613Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.692028174Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.692059144Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.692208108Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.692374152Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.692507345Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69269095Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693025029Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693316396Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693356737Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693406708Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693419078Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693446819Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69349816Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693535871Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693584242Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693619273Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693661444Z namespaces/openshift-network-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693730646Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693783677Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693823938Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.693872919Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69389896Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694010113Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694078335Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694149476Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694193107Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694237808Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69430157Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694336041Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694385032Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694415673Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694493695Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694538636Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694576877Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694648449Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694722791Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694733861Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694771642Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694815083Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694853104Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.694975507Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695048289Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695205003Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695274744Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695458569Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695542951Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695691255Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695778997Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695806098Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69589824Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.695951761Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696000162Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696035873Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696074764Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696125425Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696162646Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696203297Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696242938Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.6963151Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696391692Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696432643Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696471494Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696519325Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696558006Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696604987Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696643108Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69671746Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696748461Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696795122Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696801502Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696833393Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.696985847Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.697158731Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.697297004Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.697494769Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.697826118Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698038453Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698047313Z namespaces/openshift-network-operator/pods/iptables-alerter-747dl/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698067514Z namespaces/openshift-network-operator/pods/iptables-alerter-747dl/iptables-alerter-747dl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698139085Z namespaces/openshift-network-operator/pods/iptables-alerter-747dl/iptables-alerter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698146696Z namespaces/openshift-network-operator/pods/iptables-alerter-747dl/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698149966Z namespaces/openshift-network-operator/pods/iptables-alerter-747dl/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698182566Z namespaces/openshift-network-operator/pods/iptables-alerter-747dl/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698264929Z namespaces/openshift-network-operator/pods/iptables-alerter-747dl/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69833453Z namespaces/openshift-network-operator/pods/iptables-alerter-747dl/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698361331Z namespaces/openshift-network-operator/pods/iptables-alerter-htzfh/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698404922Z namespaces/openshift-network-operator/pods/iptables-alerter-htzfh/iptables-alerter-htzfh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698466964Z namespaces/openshift-network-operator/pods/iptables-alerter-htzfh/iptables-alerter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698473624Z namespaces/openshift-network-operator/pods/iptables-alerter-htzfh/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698479094Z namespaces/openshift-network-operator/pods/iptables-alerter-htzfh/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698507215Z namespaces/openshift-network-operator/pods/iptables-alerter-htzfh/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698585507Z namespaces/openshift-network-operator/pods/iptables-alerter-htzfh/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698649188Z namespaces/openshift-network-operator/pods/iptables-alerter-htzfh/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698702859Z namespaces/openshift-network-operator/pods/iptables-alerter-srxml/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698760201Z namespaces/openshift-network-operator/pods/iptables-alerter-srxml/iptables-alerter-srxml.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698821913Z namespaces/openshift-network-operator/pods/iptables-alerter-srxml/iptables-alerter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698829133Z namespaces/openshift-network-operator/pods/iptables-alerter-srxml/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698833633Z namespaces/openshift-network-operator/pods/iptables-alerter-srxml/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698861503Z namespaces/openshift-network-operator/pods/iptables-alerter-srxml/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.698947145Z namespaces/openshift-network-operator/pods/iptables-alerter-srxml/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699007687Z namespaces/openshift-network-operator/pods/iptables-alerter-srxml/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699039708Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699081489Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.69912086Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699159851Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699206322Z namespaces/openshift-node/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699215632Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699220562Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699249293Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699388916Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699557061Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699710875Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.699896829Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700133975Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70034034Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700376361Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700442223Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700473003Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700517815Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700547365Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700664468Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700771861Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700838763Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700877864Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700916994Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.700961126Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701000907Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701082338Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70112582Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701199101Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701271653Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701315094Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701351785Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701443327Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701524389Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701613732Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701713734Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701791036Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.701904969Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702086554Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702153935Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702200966Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702238627Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702283968Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702323809Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70236121Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702433542Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702505864Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702555145Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702591976Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702638087Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702687938Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702763151Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702801941Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702869563Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702875833Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.702904544Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.703051138Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.703221502Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.703356335Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70353934Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.703792896Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.703994721Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704032512Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704083463Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704124954Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704179406Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704207866Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704254788Z namespaces/openshift-operators/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704262478Z namespaces/openshift-operators/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704268688Z namespaces/openshift-operators/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704294458Z namespaces/openshift-operators/coordination.k8s.io/leases/sail-operator-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.7043437Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70435152Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704398221Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704538535Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704730099Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.704863803Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.705055577Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.705357415Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70557061Z namespaces/openshift-operators/operators.coreos.com/installplans/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.705599061Z namespaces/openshift-operators/operators.coreos.com/installplans/install-6j6q7.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.705988991Z namespaces/openshift-operators/operators.coreos.com/installplans/install-7jdt9.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706301898Z namespaces/openshift-operators/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706332469Z namespaces/openshift-operators/operators.coreos.com/operatorconditions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70638834Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706422371Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706477163Z namespaces/openshift-operators/operators.coreos.com/subscriptions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706510063Z namespaces/openshift-operators/operators.coreos.com/subscriptions/servicemeshoperator3.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706569365Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706621066Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706694008Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706827701Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706877033Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.706912193Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707085578Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707156729Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707229341Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707274172Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707307963Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707437867Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707473968Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707518338Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707556059Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707629491Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707692763Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707738004Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707821436Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707887988Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707901068Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.707911838Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70797805Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.708009451Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.708229576Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.708344249Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.708717798Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.708870392Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.709378855Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.709457237Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.70961305Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.709661602Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.709713303Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.709806185Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.709918128Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.709956569Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710041441Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710092413Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710128483Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710176384Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710217676Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710266577Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710312598Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710342349Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710349229Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71038828Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710439271Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710476902Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710549694Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710626016Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710666807Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710739798Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71078768Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.7108238Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710902582Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710939254Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710988845Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.710995715Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711038626Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711110408Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711150489Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71118907Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711239161Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711251061Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711271602Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711414285Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71158259Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711734153Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.711947859Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712179354Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712385899Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71239684Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712447591Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovnkube-node-f4bzh.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712609945Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-node/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712621625Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712626956Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712653636Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71279006Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712855231Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712891532Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712907243Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712912603Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.712933513Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713051486Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713115638Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713153069Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/nbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713159379Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/nbdb/nbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713180219Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/nbdb/nbdb/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71322381Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/nbdb/nbdb/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713312733Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713460346Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713490687Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/northd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713498147Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/northd/northd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713501637Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/northd/northd/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713536448Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/northd/northd/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71363629Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713762304Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/northd/northd/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713790224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-acl-logging/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713795225Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713798554Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713837715Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713922908Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713974279Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713980739Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-controller/ovn-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.713984569Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71401696Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.714303647Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.714360999Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.714375749Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovnkube-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.714379979Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.714386159Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71443159Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.715854596Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.715917947Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.715957568Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/sbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.715964678Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/sbdb/sbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.715970288Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/sbdb/sbdb/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.715990029Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/sbdb/sbdb/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716117862Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716187334Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-f4bzh/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716224915Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716300247Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovnkube-node-r4jhk.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71643843Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-node/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71644571Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.7164492Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716481061Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716590914Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716655116Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716700027Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716711107Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716714787Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716746528Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71684802Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716911702Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716937323Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/nbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716941223Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/nbdb/nbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716948993Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/nbdb/nbdb/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.716986314Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/nbdb/nbdb/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717073476Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717138308Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717178398Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/northd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717186239Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/northd/northd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717190469Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/northd/northd/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71721317Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/northd/northd/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717318112Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717382614Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/northd/northd/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717422465Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-acl-logging/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717431425Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717435755Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717457416Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717544568Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717603889Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71763999Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71764677Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-controller/ovn-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71765015Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717692321Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.717939628Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.718002549Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71803727Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovnkube-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71804579Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.71804994Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.718076311Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719394984Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719447525Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719490856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/sbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719500256Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/sbdb/sbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719504536Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/sbdb/sbdb/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719523207Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/sbdb/sbdb/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719615899Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719700901Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-r4jhk/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719800984Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.719865305Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovnkube-node-rkdkp.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720012089Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-node/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720020659Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720024859Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72004896Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720141342Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720213254Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720246905Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720253555Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720257105Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720287456Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720387468Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72045169Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720487821Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/nbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720494021Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/nbdb/nbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720497541Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/nbdb/nbdb/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720532482Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/nbdb/nbdb/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720621384Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720708456Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720740667Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/northd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720749207Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/northd/northd/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720752637Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/northd/northd/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720785038Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/northd/northd/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72088035Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720943112Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/northd/northd/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720984673Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-acl-logging/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720994393Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.720999993Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721028784Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721110256Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721159368Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721165428Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-controller/ovn-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721169048Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721217229Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721532877Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721600379Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721634349Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovnkube-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72164088Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72164438Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.721695441Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723169627Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723234359Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723256369Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/sbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72326086Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/sbdb/sbdb/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72327002Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/sbdb/sbdb/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723309331Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/sbdb/sbdb/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723398583Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723461155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rkdkp/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723482515Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723529106Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723579548Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723588128Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723615778Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723732551Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723866945Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723911886Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.723955667Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724030529Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72410499Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724149972Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724189653Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724233784Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724239974Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724243404Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724278935Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724326076Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724332356Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724363117Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.724543692Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72487919Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.725031364Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.725225648Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.725479795Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.725778382Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.725804343Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.725899895Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.725931146Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.725980577Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726022638Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72609946Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726198133Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726287455Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726328386Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726388797Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726407278Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726468359Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726528921Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726576512Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726611623Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726708845Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726758456Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726765457Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726801168Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726849609Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72689555Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.726998182Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727075984Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727271549Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727348681Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727464454Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727594617Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727785202Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727836463Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727889055Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727941686Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.727959276Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728016768Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728040198Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72811039Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728191312Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728238523Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728268214Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728322555Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728365007Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728404017Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728437598Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728487139Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72849699Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.72852772Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728666744Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728859579Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.728986972Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729182667Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729416002Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729614538Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729624438Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7ccbfd59dc-42zqt/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729654219Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7ccbfd59dc-42zqt/service-ca-operator-7ccbfd59dc-42zqt.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729756961Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7ccbfd59dc-42zqt/service-ca-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729768561Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7ccbfd59dc-42zqt/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729771961Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7ccbfd59dc-42zqt/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729796032Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7ccbfd59dc-42zqt/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.729964286Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7ccbfd59dc-42zqt/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73013059Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730174152Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730210242Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730253633Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730289554Z namespaces/openshift-service-ca/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730340825Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730383607Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730422398Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730476029Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730596422Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730695734Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730802597Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730892119Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73092585Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.730970281Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731012622Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731058863Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731124395Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731170646Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731211837Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731287079Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73132925Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73133629Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731370951Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731419992Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731460253Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731561226Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731631258Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731771081Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731845993Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.731956786Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732090899Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732254373Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732297554Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732340395Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732383336Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732428817Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732464768Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732504469Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732582701Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732658593Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732724965Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732771056Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732810087Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732845518Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732888979Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.732896199Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73292907Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.733079764Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.733252998Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.733381491Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.733567916Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.733875623Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734068888Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734076478Z namespaces/openshift-service-ca/pods/service-ca-7b994bc59-nstd9/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734105479Z namespaces/openshift-service-ca/pods/service-ca-7b994bc59-nstd9/service-ca-7b994bc59-nstd9.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734179131Z namespaces/openshift-service-ca/pods/service-ca-7b994bc59-nstd9/service-ca-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734186731Z namespaces/openshift-service-ca/pods/service-ca-7b994bc59-nstd9/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734190951Z namespaces/openshift-service-ca/pods/service-ca-7b994bc59-nstd9/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734218232Z namespaces/openshift-service-ca/pods/service-ca-7b994bc59-nstd9/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734444398Z namespaces/openshift-service-ca/pods/service-ca-7b994bc59-nstd9/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734507799Z namespaces/openshift-service-ca/pods/service-ca-7b994bc59-nstd9/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73453204Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734572081Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734621382Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734654873Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734736015Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734775346Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734830007Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734870868Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.734915959Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735031092Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735102244Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735178266Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735248647Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735301539Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73533552Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735379421Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735420692Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735493184Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735533215Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735576336Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735648988Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735717379Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73575794Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735855763Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.735922554Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736005436Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736072008Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73614694Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736271783Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736432797Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736484118Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736526849Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736575331Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736612632Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736651562Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736719044Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736790126Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736869618Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736911139Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73695314Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.736991561Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.737023662Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.737075853Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.737084753Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.737111674Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.737256067Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.737427002Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.737560505Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.737790461Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738039377Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738295013Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738329874Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738380765Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738405476Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738460927Z namespaces/openshift/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738500768Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738550409Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738621031Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738668312Z namespaces/openshift/apps/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738732444Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738826356Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.738896748Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73896931Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739018421Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739049922Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739096213Z namespaces/openshift/batch/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739125624Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739198225Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739244977Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739285368Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73936455Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739401821Z namespaces/openshift/core/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739438871Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739536154Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739601266Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739701078Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.73978135Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739862862Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.739978705Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.740154579Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.740200141Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.740241962Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.740283863Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.740320453Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.741595635Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.741620286Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.741743029Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.741823501Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.741928424Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.742025566Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.742109298Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.742269002Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.742428236Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.742548539Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.742639971Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.742768414Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.742883177Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.74298982Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.743119133Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.743197315Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.743279047Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.74338168Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.743491622Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.743600255Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.743721138Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.74381618Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.743901762Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.743997765Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.744085847Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.7441889Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.744278082Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.744375214Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.744483277Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.744580119Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.744692832Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.744816395Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.744925688Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.74500558Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745096282Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745193145Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745305767Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745449551Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745528963Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745633956Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745764219Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745869742Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.745998575Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746087007Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746180919Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746299602Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746421085Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746524818Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746657631Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746783514Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746870476Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.746955768Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747076351Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747195684Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747315997Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.74743265Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747542153Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747628155Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747752528Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747864441Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747944103Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.747985494Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748073466Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748144538Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748186889Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.74822777Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748272361Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748310792Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748354193Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748363423Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748396944Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748541198Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748823655Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.748955898Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749170603Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749412219Z namespaces/openshift/operators.coreos.com/clusterserviceversions/servicemeshoperator3.v3.2.0.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749614395Z namespaces/openshift/policy/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749653236Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749727207Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749756908Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749805449Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749813489Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.74984921Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.749956643Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.750075866Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.750178069Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.750284711Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.750458705Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.750562668Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.750696881Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.750827725Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.750966478Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.751064811Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.751173703Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.751299046Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.75143475Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.751535872Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.751656255Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.751782258Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.751881871Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752003894Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752091556Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752187938Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752287281Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752388763Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752490556Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752583368Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752700911Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752809304Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.752905496Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.753016959Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.753195183Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.753294466Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.753398028Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.753494421Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.753614634Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.753773648Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.753981813Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754075705Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754180148Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754296131Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754391463Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754493696Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754598688Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754738692Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754865255Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.754977238Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755096581Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755239074Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755389188Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755496501Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755605813Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755690646Z network_logs/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755750207Z network_logs/cluster_scale [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755820159Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755902241Z network_logs/multi-networkpolicy [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.755971152Z network_logs/net-attach-def [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.756042924Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.756186118Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.75627304Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.758368922Z nodes/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.758396802Z nodes/debug [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.758491215Z nodes/ip-10-0-128-226.ec2.internal/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.758533436Z nodes/ip-10-0-128-226.ec2.internal/cpu_affinities.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.758973377Z nodes/ip-10-0-128-226.ec2.internal/dmesg [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.759287455Z nodes/ip-10-0-128-226.ec2.internal/ethtool_channels [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.759361127Z nodes/ip-10-0-128-226.ec2.internal/ethtool_features [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.759443929Z nodes/ip-10-0-128-226.ec2.internal/ip-10-0-128-226.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.759994002Z nodes/ip-10-0-128-226.ec2.internal/irq_affinities.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.760149076Z nodes/ip-10-0-128-226.ec2.internal/lscpu [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.760275019Z nodes/ip-10-0-128-226.ec2.internal/lspci [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.760366102Z nodes/ip-10-0-128-226.ec2.internal/podresources.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.760474054Z nodes/ip-10-0-128-226.ec2.internal/pods_info.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.760595927Z nodes/ip-10-0-128-226.ec2.internal/proc_cmdline [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.76069271Z nodes/ip-10-0-128-226.ec2.internal/sysinfo.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.762341521Z nodes/ip-10-0-128-226.ec2.internal/sysinfo.tgz [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.762473434Z nodes/ip-10-0-128-243.ec2.internal/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.762515065Z nodes/ip-10-0-128-243.ec2.internal/cpu_affinities.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.762869544Z nodes/ip-10-0-128-243.ec2.internal/dmesg [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.76311261Z nodes/ip-10-0-128-243.ec2.internal/ethtool_channels [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.763182232Z nodes/ip-10-0-128-243.ec2.internal/ethtool_features [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.763261334Z nodes/ip-10-0-128-243.ec2.internal/ip-10-0-128-243.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.763657373Z nodes/ip-10-0-128-243.ec2.internal/irq_affinities.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.763830857Z nodes/ip-10-0-128-243.ec2.internal/lscpu [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.763955071Z nodes/ip-10-0-128-243.ec2.internal/lspci [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.764044383Z nodes/ip-10-0-128-243.ec2.internal/podresources.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.764152056Z nodes/ip-10-0-128-243.ec2.internal/pods_info.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.764281179Z nodes/ip-10-0-128-243.ec2.internal/proc_cmdline [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.76434867Z nodes/ip-10-0-128-243.ec2.internal/sysinfo.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.766113224Z nodes/ip-10-0-128-243.ec2.internal/sysinfo.tgz [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.766281429Z nodes/ip-10-0-141-25.ec2.internal/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.76632587Z nodes/ip-10-0-141-25.ec2.internal/cpu_affinities.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.766663938Z nodes/ip-10-0-141-25.ec2.internal/dmesg [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.766970446Z nodes/ip-10-0-141-25.ec2.internal/ethtool_channels [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.767033197Z nodes/ip-10-0-141-25.ec2.internal/ethtool_features [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.767114359Z nodes/ip-10-0-141-25.ec2.internal/ip-10-0-141-25.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.767483668Z nodes/ip-10-0-141-25.ec2.internal/irq_affinities.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.767635182Z nodes/ip-10-0-141-25.ec2.internal/lscpu [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.767786366Z nodes/ip-10-0-141-25.ec2.internal/lspci [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.767870428Z nodes/ip-10-0-141-25.ec2.internal/podresources.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.76797346Z nodes/ip-10-0-141-25.ec2.internal/pods_info.json [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.768089743Z nodes/ip-10-0-141-25.ec2.internal/proc_cmdline [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.768161065Z nodes/ip-10-0-141-25.ec2.internal/sysinfo.log [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.769837367Z nodes/ip-10-0-141-25.ec2.internal/sysinfo.tgz [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.770026682Z pod_network_connectivity_check/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.770074543Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.770174115Z static-pods/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.770180746Z static-pods/kube-apiserver/ [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.779328353Z [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.779345073Z sent 78,911 bytes received 9,925,902 bytes 6,669,875.33 bytes/sec [must-gather] [must-gather-hlkzt] OUT 2026-06-15T07:36:38.779349093Z total size is 120,965,338 speedup is 12.09 [must-gather] [must-gather ] OUT 2026-06-15T07:36:39.292177565Z namespace/openshift-must-gather-d7tsl deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 52b23002-b21e-42e1-a029-20bb4a09421e [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.20" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-vgn9g [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-vgn9g-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-vgn9g [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 19666606 Jun 15 07:38 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-vgn9g/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts c844272] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-vgn9g/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] f1c1862..c844272 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-06-15T07:38:40.579Z","type":3}]